Stealth is fun.
Shadowdancer may be one of the most popular Prestige Classes in 3.5e, and that is solely due to the Hide in Plain Sight feat. Many players enjoy the thought of sneaking invisibly to the enemy and rolling insane backstab/sneak attack damage.
Unfortunately, stealth in D&D is not always that fun.
Now, the backstab part is awesome, and that's why most stealth players enjoy it. The problem is that the D&D mechanics as they are played out in most campaigns do not make much of stealth beyond a canned skill challenge. By looking at some good stealth games for the computer, such as Dishonored or Assassin's Creed, we can take some tips and add them to our campaigns.
Objectives
This is the biggest change that a DM has to foster in his campaign. As mentioned before, the objective of stealth is almost always just to get in some extra sneak attack damage. Stealth gets boring when, in the end, it's only about combat. There is nothing wrong with sneak attacks, of course. Some of the most memorable moments in my campaigns have been sneak attacks (double crit + 4x backstab damage FTW?), but stealth needs variety.
The purpose of stealth is to remain undetected. Let stealth be a tool for defeating encounters. If the players successfully sneak around an entire group of hobgoblins, give them full XP as if they had beaten the fight. And don't just stop there. If you want great stealth encounters, turn it into a real challenge like Dishonored does. Make enemies move around somewhat unpredictably. Have your players use distractions, or find opportunities to pick off the enemy one at a time. Give them bonus XP or a better reputation for being able to complete encounters without bloodshed, similar to Dishonored. Also like Dishonored, make a few combat encounters really dangerous if you rush right into them, and be sure to make that fairly clear through in-game information.
Environment
The world is bigger than a grid. Description helps. Open up the terrain for movement, like Assassin's Creed. Let them sneak past the royal guard by balancing across the rafters of the great hall or by sneaking over the rooftops to bypass the thugs waiting for them in the street. Think in 3D even though the grid is 2D.
Light is a huge factor for stealth in a lot of games, such as Amnesia: The Dark Descent. It ought to be very important in D&D as well, what with all the torches, lanterns, and magical lights often found in its environments. Have players make strategic use of light. One campaign, my players doused a torch while the guard was on the other side of the building so that when he came back, he couldn't see them sneaking inside. Unfortunately, the sudden lack of light alarmed him, which leads to another point:
The Chase Sequence
The way you describe the ninja character as cycling through backstab -> run -> hide -> backstab definitely confirms this as a bad pattern of stealth. One of the biggest flaws of the first Assassin's Creed game was how you could stab someone, run like heck, hide on a bench right around the corner, then go back and stab someone else. Rinse. Repeat. Worst of all, until they introduced notoriety in later games, it seemed like everyone forgot what you did.
Dishonored is a much better example of how to do detection and chase effectively. On the very first detection, the enemy is immediately alert and aware of the fact that you do not belong here. Hostility begins right away, and the chase is brutal. In a chase, NPCs do not let you get away unless you do something really daring. In the TV Show Burn Notice, the main character remarks during a narrative in a chase sequence that the only way to escape a chase is to do something that the people chasing you won't do -- like jumping off a roof.
And even if you get away, the NPCs should not just "forget" about you. They should be on high alert until you die or they are convinced that you have been driven off. Enemies on high alert for a stealth PC should not be easy to catch off-guard. In addition, they should not be splitting up alone if they are even reasonably intelligent.
Have NPCs adopt tactics like the PCs tend to act when encountering stealthy foes.
All Alone
"But I'm the only stealth character on the team!"
This is roughly the ninja player's position, I take it. I've been there. Fortunately, you don't have to be reliant on stab-and-run to be useful. A number of the former tips are intended for stealth-based encounters, but here's what a stealth PC has gotta do to have fun with stealth while your allies are kickin' down doors in the name of Tempus:
1: Wait for the encounter to get started. Be out of sight on the periphery.
2: Sneak up to a squishy target.
3: Stab.
4: ???
5: Profit.
An ultra-stealthy character is ideal for taking out priority targets. Then, using other skills, such as acrobatic-type skills, make a daring escape. Not just running away by pure movement points, but dodging between pillars, leaping onto ledges, or tumbling past enemies to rejoin your allies.
Stealth should get you into the fray. Speed and tricks should get you out.
First, this adds variety to your actions as a stealthy character.
Second, it should be hard to lose detection when enemies are tracking your movements so closely.
Beyond combat, a stealthy character can still be a great asset. Perhaps you can open a gate while the party is fighting. Maybe you can sneak into a camp and rescue a prisoner while the party is attacking from the opposite side. Generally, you should avoid going too lone wolf unless your party wants you to do so, because that's dangerous and slows down the game for others. Performing a stealth mission while the party fights a battle has been the best possible scenario in campaigns I have played. It keeps everyone busy, provides a distraction, and lets your group benefit from stealth simultaneously.
TL;DR version: Stealth should be more than dice rolls. It needs to interact with the environment and the intelligence of the NPCs involved. It should be rewarding, fast-paced, and require cleverness more than just sneakiness. When done right, it should give the party big advantages as a whole.
Energy
Put focus into roleplaying, even if you're watching someone else do it. Focus yourself on what they are saying and doing, even if it's kinda boring, and project your body language and voice while you're acting in-character. Be much less high-intensity when simply describing your bonuses while rolling, or asking someone to pass the chips. This will create natural focus on the roleplaying aspect of the game.
Speak In-Character
Wherever possible, assume what others are doing is a prompt for in-character roleplaying. DM asks what you do? Turn to someone, and say 'Philius, methinks we should cross that bridge and by Everam and St George, charge those there gnolls with swords in hand. Once we have them subdued, we'll take some answers from them, so we must leave at least some alive! What say you, well-met friend?"
Even if people don't respond in-character, and instead shift it back ooc, roleplaying just happened. Keep doing it and others will soon follow.
Ham It Up
Your character is 'quiet' and 'reasonable' and NOPE. Your character is the hammiest of ham. He's a loud cliche. He is instantly identifiable as the tropes that make him up - and he defines the setting by his very presence. It's unfortunate - but humans love ham. They love it.. a lot. Be something simple and understandable and loud, and they will get with the program really fast.
You can do this by being a masterful actor and roleplayer with any character, even a non-hammy one, but it is easiest with ham, so ham I will advise. Your paladin isn't just a paladin who likes cheese and moonlit walks on the beach - he's SIR GALAHAD THE MIGHTY, SUBDUER OF THE PEASANTS, DEFENDER OF THE WOMENFOLK, AND HIS MOUSTACHE BRISTLES AT THE SLIGHTEST SUGGESTION OF DRAGONS.
'Big' traits tend to focus things on the roleplaying a lot faster. Simpler is easier for the audience to understand.
Find Allies
Find people who will respond to your dramatic offers. When you address people, address them first, so they respond in-character, and then immediately pull other people in. People ignore offers initially, but if something is already rolling, they'll get rolled in with it. Some people will instinctively resist roleplaying offers, for all kinds of reasons - learn to identify them too, and offer to them last, once the roleplaying scene has the most momentum.
Be good at plot
Being able to identify where the adventure is going will let you advance the plot during a roleplaying scene - which both speeds up the adventure and means the time spent on roleplaying won't cause a weak GM to not let you hit the end of it.
Roleplay during combat
'LOOK OUT, FARAMIR! THE GNOLL IS AT YOUR BACK!' 'Galahad charges at the gnoll attacking' moving mini 'faramir, and' rolls dice 'swings at it with his mighty sword.' By including both speech and roleplaying-description in amongst your mechanical actions, you partially negate the disconnect that happens during the mechanics-rich combat portions of sessions. Have to know what you are doing on your turn before your turn rolls around, or anti-roleplayers will complain your roleplaying is slowing things up if you are not clearly doing it faster than anyone else.
Additionally, being good at combat, and giving tactical advice in-character that leads to defeating enemies quickly, will give more time overall for non-combat-constrained roleplaying itself.
Occasionally, roleplay during others' turns - have Galahad shout an encouraging phrase at an opportune moment. This has to be rare, and well-timed, though - an advanced technique.
Be Heroic, or Dastardly
Again, ham. By being heroic, and roleplaying it hard, you make other people who are not roleplaying feel heroic. By being dastardly, and roleplaying it hard, you make other people feel heroic also who are not roleplaying. You're giving them some of your roleplaying energy in a way that feels good for them. Morally grey is, again, a tougher sell. Note this isn't 'good' or 'evil', it's more saturday morning cartoon than that. Snidely Whiplashi, or Dudley DoRight.
Incorporate the GM
Don't just roleplay at fellow players. Roleplay at NPCs. Treat them with importance, and give the GM offers to roleplay right back at you. All of this applies to the GM, too. Getting the GM on-board with roleplaying, especially if you can advance the story while doing so, will be a tremendous boon to your cause.
You're right, by the way. Premade adventures, split up groups, schedules, public venues, this stuff just kills roleplaying and really makes it quite hard - I literally could not design a better system to do so.
But even in those kind of circumstances, I have personally sparked roleplaying in some extremely tough crowds. You won't see a huge improvement - but even the tiniest bit of roleplaying can be a huge welcome to you if you're in a roleplaying drought, and if you play with regularly the same pool of people, you'll find people gravitating to you that appreciate roleplaying, perhaps even to the extent that people will fight to have you in their groups.
Overall, though, the roleplaying will be in many ways a simpler thing than the rare high level roleplaying you can get in a home group.
But it's certainly not impossible.
Just have the courage to keep trying and don't give up.
Best Answer
There are a number of tactics but it boils down to creating situations where the other actions are necessary to continue to fight effectively.
Here are some examples.
The Dodge action:
"On the far end of the room is a large repeating crossbow attached to the wall set to fire on you as you approach. The cultists are continuing the ritual, daring you to cross the runes before you. Everyone roll Initiative. [sets up Initiative order]"
"Can I identify the runes?"
"Roll Arcana. [Sees 24 result]. You know these runes are proximity runes likely to set off the crossbow when you pass them."
"Alright, I take the Dodge action so the bolts will have a harder time hitting me and cross with my weapon drawn."
The Disengage Action
"As the archdevil starts to bleed from the beating you've given it, it glows a bronze color and your weapons start to bounce off of it. At the same time fire bursts out of the fiend. Everyone in 10 feet, make a Dexterity saving throw."
[Rolls are made] "You all avoid damage for now but the bronze aura has only faded a bit, and small flames start to appear on his skin once more. Your turn."
"I use Disengage to get away from the fire and duck behind the altar until the bronze thing is gone."
The Help action
(This one is already very useful in situations where a person has run out of resources like spell slots and can grant advantage to a more useful ally, or to cancel out disadvantage for your big damage dealers)
"In this magical fog, you can't quite make out the enemy. You all have disadvantage on your attacks."
"I run up to where I hear the enemy move and use the Help action to advise my ally where to shoot his bow."
Show don't tell
If they still don't use these tactics in situations that warrant it, have an NPC demonstrate their effectiveness with them (either an allied NPC or an enemy if possible).
Sometimes players are just ignorant of how effective other types of action can be at no fault of their own.