Many people are discovering this problem and very clearly the easiest method is going to be to try to work with people who are on the same quest as you.

Given that will not always be possible the choice comes down to helping your friend catch up or speeding your friend to your quests and allowing them to replay them in their own time.

By maintaining a large friend list by joining a clan, playgroup or adding your name to the spreadsheet of a forum you participate in, you should be able to find someone who is in the same general vicinity as you. Unfortunately there is no method at present to group friends or to edit their names into something more familiar to you (eg a forum ID that you recognize instead of their BattleTag/RealID) so you should maintain a list somewhere to help you recall.

A great number of people (myself included) are deciding to solo Normal mode so that they can take all the time in the world to experience all the details and nuances before worrying about co-op play.

With more experience I've also taken the following action; play one character as my "get through the game, unlock things and get good loot" character and started another that I quickly levelled up. In my case my primary is a demon hunter and my secondary a barbarian. I get to play different content but still get to keep pace with my friend.

I also installed skype to give us voice chat, which helped a lot.

I defeated Diablo as a L51 Demon Hunter last night, it was a tricky kill, as you correctly point out the Shadow Realm is one of the harder parts due to the lack of health restore points. Unlike some of my answers which are based on second hand knowledge, this is direct experience. Your milage may vary :)

Here is my general build and strategy, I have tried many other things that did not work well.

The key factors here are slowing Diablo down, avoiding damage yourself and finding ways to recover health as you go.

I'm using dexterity to help with damage (943% increase due to dex), 2k armor and 4k dps overall with bonuses included.

Slowing

For my primary damage dealing I chose to use rapid fire with web shot. This skill plays well with +Life on Hit and with Life Steal. Also having some +Hatred Regeneration equipment helped keep this going as long as possible. I also use a Raven Companion for a little supplementary damage and regeneration.

I chose entangling shot due to its hatred regeneration. You can choose to use the heavy burden rune to increase the slowing time as I did or you can use justice is served to recover hatred more quickly and switch back to rapid fire.

Avoiding Damage

It should be needless to say but you should be kiting quite handily by A4 NM. In order to keep kiting Diablo you need to slow him down.

I've been using a shield to help bump my armor numbers up but the block and dodge bonuses I received were really the saving grace. I have around 100 fire/lightning resist and around 50 in the rest (not astronomical numbers but every bit is helping). I found that in the main world fights that the lightning/fire burst could quickly drop me but it was usually his circle of fire that hit me hard. To avoid these sudden situations I found that Vault with the Rattling Roll rune was invaluable. Not only could I survive at close range until an attack was imminent, but I could leap directly towards Diablo and add some stun in. I also used this to avoid the bone cages. If I found that I needed to run, I could easily follow up with a Smoke Screen with the Lingering Fog rune after he shook off stun.

In the Shadow Realm I found that if I waited until my clone was just spawning and hit Smoke Screen I could usually wipe him out with Rapid Fire, hit Smoke Screen again, collect the health globe and reposition for Shadow Diablo very effectively. Oddly I found my clone dealt far more damage than Shadow Diablo, especially when being effective at kiting.

Recovering Health

For the shadow realm this is tricky. As I pointed out above, you can use vault with stunning and smoke screen to get you to the dropped health globes, if you have any equipment buffing the health bonus from that, great. I have a base of about 12,000 hp to work with, and I use +Life on Hit (~100) and some %Damage to Life (3%) to keep myself going. I also regenerate about 100 Life/Second. With this setup I can run away, heal a little, throw some quick shots in and slowly heal myself back up.

I also made sure to equip some resplendent health potions (6500 life). Some times that extra 2k kick over heroic potions made the difference. For emergencies I also kept Preparation with Battle Scars available, a missed vault or smoke screen with low health can be fatal. When I started scrambling away I'd rather hit vault a couple of times, preparation and then smoke screen to get out of harm's way.

I used the Cull the Weak, Sharpshooter and Archery passive skills to add some kick my attacks and the were noticeably helpful.

Using this setup it took me 2 tries to kill Diablo and the 2nd try went very smoothly.