To evoke a sense of danger, three things are needed:
- a knowledge of danger existing
- an uncertainty that it is an escapable danger
- a certainty that a strongly undesired outcome is possible
So, you need to forshadow the existence of the danger, make it uncertain that they can overcome it successfully, and remind them by the situation that it's possible that they will not be happy with all available outcomes.
In practice, in D&D, things like asking what their current HP are, and then making a quick calculation, and going "Ooh... crunchy!" can up the sense of danger.
Getting the outcome range to include strongly undesired is tricky, tho'. The player needs to have a strong sense of buy-in on their character to actually be overly worried about the loss of the character. I've found that the longer the character generation, the more buy-in is present at start.
For example, a Rolemaster character at level 1 is about equally weak as a Cyclopedia D&D character, and the RM character takes about 30-90 minutes (depending on the player) to roll up, while the Cyclopedia character takes 5-15 minutes. The RM character hurts way more as a fatality in level 1 than the D&D character, and is just as much cannon-fodder.
So, with ways to make it matter more are more choices made in Character Generation (CGen), more significant choices during CGen, more involved CGen, more in-character play, and more personal impact on the character in play. All of these make loss of the character more profound, and thus more undesirable.
Once achieved, reveal the danger as per the other question, and make certain that players realize character loss isn't off the table... and genuine concern, even a true sense of danger, naturally occurs.
If your players are easily frustrated by a few bad dice rolls, that's a problem with your game in general. Bad dice rolls happen. And they will happen a lot. If the laws of the universe don't change in the near future, I would even dare to say they will happen with the same frequency that applies to good dice rolls.
After all, your players don't get frustrated rolling too good, do they? Did they ever leave the table because the rolled 3 criticals in a row totally dominating the encounter? So the real problem probably is, that their only action and therefore their only fun producing aspect of the game is winning the dice roll. That is fine for a board game, but not exactly the goal of roleplaying games.
You can do a lot more in a turn of combat than stand there and hit the enemy. You can move to flank the enemy. You can taunt him (just in character, without any rules involved). You can be creative. You can have a lot of fun with failures. My most memorable moment in roleplaying was when my ninja character failed so miserably sneaking out of a bar that she ended up on the doorsteps with a broken ankle and screaming. Sure, that was a failure, the worst possible combination of dice I ever saw, but it was still fun and a happy memory meeting the people of this group even ten years later. But you need to encourage it and you need to allow it, even if (especially if!) it's not in the rules.
Example: Playing a specialized magic user, our party once met a monster that was completely immune to any of my magic. I could have taken the second (or third) row in combat shooting ordinary arrows at it. I would probably have missed 20 times in a row. It would have been incredibly boring. After 10 misses I would probably have left the table, too. Life is too short to be bored. Instead my character switched into light armour (no proficiency, but who cares), got a dark cape and a dagger and sneaked behind the monster. He failed the sneak roll, he was unable to cast magic due to the armor and he really sucked at hitting it with the dagger. BUT: the GM decided it would turn to me and leave it's back turned to the warrior leaving it open to his attacks, because even the dumbest monster knows that those sneaky dark dagger people hurt the most. I failed every single roll that combat and still contributed and had fun.
Remember, as a GM you need to encourage and allow it. No rule ever said that monsters need to turn to the sneaky git with a dagger. That was good GMing. We had fun. Much more fun than any fireball-damage-dice-rolling-spell could have brought. Build your encounters so that the players feel they can be creative. And allow them to be creative. If the players are creative, then no dice roll is needed to "allow" them to contribute. Contribution is measured in fun, not damage points.
--- Edit: ---
The first part was about what can be changed in the gaming group and playstyle to make failed rolls less frustrating. A commenter remarked that sometimes it has nothing to do with the game or group and that's absolutely right:
Some players simpy cannot lose. They can't. If they lose, they get frustrated. But losing is part of any game. I guess we all know people who are like that. It's also never them. It's the noob team. Or the dice. Or whatever. If the game was lost, somebody must have been a bad player. That you can play a game, be good at it, have fun and still lose doesn't fit in their world view. For them, it's about winning, not about having fun playing. If this is the case with your player, there is little that you can do to change that. For him to be the winner every time, the rest would have to lose. And you can't have that in a group. That's not fair to the others.
Some players are just bad at statistics. They don't have bad luck. They only feel like they had bad luck because they don't know better. Some people need a 18 to hit on a d20 and think missing 3 times in a row is incredibly bad luck. It's not. To the contrary, it would have been pretty lucky to hit just once in three tries. Make sure the players know at least basic statistics and can convert your systems dice rolling to percentages, so they have a number they can grasp.
Some people are building their characters with damage dealing as a priority. They can dish out huge amounts. Many dice. Large numbers. But most systems are quite balanced, so to achieve this, they sacrificed something. Most likely their chance to hit. So they sit there round after round waiting for their one moment of glory where they land a lucky hit to show off their uber damage. That's a decision. They could as well have build a character that attacks three times a round for little damage and in a good system it would have the same end result. As long as they look only at one number (damage) and not at the full picture (damage * chance to hit), they will frustrate themselves every time. Make sure they know that this is their own decision, because they can only change this situation themselves.
Best Answer
Make Things Personal
[Since the problem seems to be that the players are wrapped up in the mechanics - this advice is meant to be system/version neutral.]
If you want to increase engagement with something other than the dice, make the situations personal to the characters and by extension, the players.
Give the characters individual motives - each has an individual quest or two. And they may be partially opposed.
A great way to generate motives is to give the players preexisting relationships, as Mike Shea suggests in his blog post at SlyFlourish.com: Fiasco-Style Relationships.
Reward players for chasing their objectives and role-playing to their relationships.
An example
By request many moons ago, I took over DMing a AD&D Dragonlance Campaign late in the series. The previous DMs had treated it as a dungeon crawl and many of the players were bored.
Riverwind and Goldmoon [the last of their tribe] were played by a real-life couple and she wasn't at all engaged. First, I threw them all in the BBEG's Prison. Then pulled her aside an told her that Goldmoon was pregnant and hadn't told anyone yet. Instantly that player became engaged with the desire to protect their unborn child.
Mechanics support
Some games have actually woven story-dependent relationships into the mechanics: Various D&D editions have Themes/Roles/etc. that specifically add motivation to a character. Such as The Neverwinter NobleDDI [in effect, Aragorn's backstory from Lord of the Rings.] Often these include flavored powers that the player can use, often out-of-combat. Even if this isn't your edition, you can steal these ideas to great effect.
Of course, the tried-and-true "looking for a family relic", "avenging my beloved's murder", and the like have always been a great simple motivations. Just drop hints here and there about these items and you'll see engagement spike.