OK, I don't have time to answer this as I want to. My background is in psychology, and I fell into role playing games when I turned 10 in 1976. So by the time I was in college, understanding where the term Roleplaying game really came from, I understood the critical nature of immersion, how it is the most important ingredient for game success.
And to be clear, the definition of immersion is to "Immerse oneself into the identity and Role of the part one is playing. To respond, as much as possible, as the person one is playing, not as oneself."
And before getting into the smaller details, I will dive right into the fact that the very system/game one chooses has a huge amount to do with the amount of Immersion.
Metagaming is the opposite of immersion. You use both terms, but I need to make that absolute definition from the beginning. This also means rules that encourage metagaming decrease the immersion in a game and therefore, decrease the main ingredient of a roleplaying game. The mechanics are called "Dissociated Mechanics", a term coined by Justin Alexander. This is very worth reading, because it gets into many of the larger picture issues with players being able to use in-game logic to see the world around them, as opposed to the rules forcing dissociation from in-game logic.
Once the players assume that rules are going to determine the content of an encounter or treasure (based on EL, or whatever) instead of what the environment or history of the area dictate, verisimilitude is lost.
Vreeg's Rules of Setting design are also heavily immersion related. My current campaign is 26 or so years old (started in '83). Building verisimilitude is a huge part of this.
Vreeg's first Rule of Setting Design
Make sure the ruleset you are using
matches the setting and game you want
to play, because the setting and game
WILL eventually match the system.
Corollary to Vreeg's First Rule
The proportion of rules given to a
certain dimension of an RPG partially
dictate what kind of game the rules
will create. If 80% of the rulebook
is written about thieves and the
underworld, the game that is meant
for is thieving. If 80% of the
mechanics are based on combat, the
game will revolve around combat.
- Multiply this by 10 if the reward
system is based in the same area as
the preponderance of rules.
2nd Corollary
Character growth is
the greatest reinforcer. The
synthesis of pride in achievement
with improvement in the character
provides over 50% of the
reinforcement in playing the game.
Rules that involve these factors are
the most powerful in the game.
Vreeg’s Second Rule of Setting Design
Consistency is the
Handmaiden of Immersion and
Verisimilitude. Keep good notes, and
spend a little time after every
creation to ‘connect the dots’. If
you create a foodstuff or drink, make
sure you note whether the bars or inns
the players frequent stock it. Is it made
locally, or is it imported? If so,
where from? If locally made, is it
exported?
Vreeg's Third Rule of Setting Design
The World In Motion is critical
for Immersion, so create 'event
chains' that happen at all levels of
design. The players need to feel like
things will happen with or without
them; they need to feel like they can
affect the outcome, but event-chains
need velocity, not just speed.
Vreeg's Fourth Rule of Setting Design
Create motivated events and
NPCs, this will invariably create
motivated PCs. Things are not just
happening, they happen because they
matter to people (NPCs). There is no
need to overact, just make sure that the
settings and event-chains are
motivated and that the PCs feel
this.
Vreeg's Fifth Rule of Setting Design
The Illusion of Preparedness is critical
for immersion; allowing the players to see
where things are improvised or changed
reminds them to think outside the setting,
removing them forcibly from immersion.
Whenever the players can see the hand of the GM - even when the GM needs to change things in their favor -
it removes them from the immersed position.
(Cole, of the RPGsite, gets credit for the term).
Remember that part of immersion is the lack of feeling walls around and rails under the characters. This means that the players should not feel that there are things that their character cannot do solely because of the rules or the GM's mindset. The job of the GM is to enable roleplay, not to inhibit it.
This also means the GM must be as immersed as the players, or more.
Another big-picture thing that may irk some folk who sell stuff is that published settings can hurt immersion. They don't destroy it; but when the players have a lot of knowledge about a setting that their character would not have, this increases the opportunity to use it, consciously or unconsciously. Similarly, if your setting has its own bestiary that the characters learn as they go along, or at least a lot of homebrew tweaks, the players get used to working with the in-house data and not trusting the published sources.
If you have done all of this larger-scope stuff, the smaller scope stuff becomes easier. As a GM with miles on the tires, I find that playing up the level of knowledge my NPCs might have and do not have helps keep the players in the same mindset. Players key heavily off the way the GM plays their NPCs. They won't do the funny voices or the mannerisms if the GM does not, and if the GM is particularly careful about what their NPCs know and don't know, especially verbally, the players emulate this.
If your players are easily frustrated by a few bad dice rolls, that's a problem with your game in general. Bad dice rolls happen. And they will happen a lot. If the laws of the universe don't change in the near future, I would even dare to say they will happen with the same frequency that applies to good dice rolls.
After all, your players don't get frustrated rolling too good, do they? Did they ever leave the table because the rolled 3 criticals in a row totally dominating the encounter? So the real problem probably is, that their only action and therefore their only fun producing aspect of the game is winning the dice roll. That is fine for a board game, but not exactly the goal of roleplaying games.
You can do a lot more in a turn of combat than stand there and hit the enemy. You can move to flank the enemy. You can taunt him (just in character, without any rules involved). You can be creative. You can have a lot of fun with failures. My most memorable moment in roleplaying was when my ninja character failed so miserably sneaking out of a bar that she ended up on the doorsteps with a broken ankle and screaming. Sure, that was a failure, the worst possible combination of dice I ever saw, but it was still fun and a happy memory meeting the people of this group even ten years later. But you need to encourage it and you need to allow it, even if (especially if!) it's not in the rules.
Example: Playing a specialized magic user, our party once met a monster that was completely immune to any of my magic. I could have taken the second (or third) row in combat shooting ordinary arrows at it. I would probably have missed 20 times in a row. It would have been incredibly boring. After 10 misses I would probably have left the table, too. Life is too short to be bored. Instead my character switched into light armour (no proficiency, but who cares), got a dark cape and a dagger and sneaked behind the monster. He failed the sneak roll, he was unable to cast magic due to the armor and he really sucked at hitting it with the dagger. BUT: the GM decided it would turn to me and leave it's back turned to the warrior leaving it open to his attacks, because even the dumbest monster knows that those sneaky dark dagger people hurt the most. I failed every single roll that combat and still contributed and had fun.
Remember, as a GM you need to encourage and allow it. No rule ever said that monsters need to turn to the sneaky git with a dagger. That was good GMing. We had fun. Much more fun than any fireball-damage-dice-rolling-spell could have brought. Build your encounters so that the players feel they can be creative. And allow them to be creative. If the players are creative, then no dice roll is needed to "allow" them to contribute. Contribution is measured in fun, not damage points.
--- Edit: ---
The first part was about what can be changed in the gaming group and playstyle to make failed rolls less frustrating. A commenter remarked that sometimes it has nothing to do with the game or group and that's absolutely right:
Some players simpy cannot lose. They can't. If they lose, they get frustrated. But losing is part of any game. I guess we all know people who are like that. It's also never them. It's the noob team. Or the dice. Or whatever. If the game was lost, somebody must have been a bad player. That you can play a game, be good at it, have fun and still lose doesn't fit in their world view. For them, it's about winning, not about having fun playing. If this is the case with your player, there is little that you can do to change that. For him to be the winner every time, the rest would have to lose. And you can't have that in a group. That's not fair to the others.
Some players are just bad at statistics. They don't have bad luck. They only feel like they had bad luck because they don't know better. Some people need a 18 to hit on a d20 and think missing 3 times in a row is incredibly bad luck. It's not. To the contrary, it would have been pretty lucky to hit just once in three tries. Make sure the players know at least basic statistics and can convert your systems dice rolling to percentages, so they have a number they can grasp.
Some people are building their characters with damage dealing as a priority. They can dish out huge amounts. Many dice. Large numbers. But most systems are quite balanced, so to achieve this, they sacrificed something. Most likely their chance to hit. So they sit there round after round waiting for their one moment of glory where they land a lucky hit to show off their uber damage. That's a decision. They could as well have build a character that attacks three times a round for little damage and in a good system it would have the same end result. As long as they look only at one number (damage) and not at the full picture (damage * chance to hit), they will frustrate themselves every time. Make sure they know that this is their own decision, because they can only change this situation themselves.
Best Answer
Sometimes, all that's needed to get shy, anxious players to open up is time. If they're more secure in their place in the group and in how their contributions will be received, they'll relax.
One big factor can be the group itself. If you've got a group where one or two people consistently dominate discussion and action, and a couple of others tend to hang back, try running a session or two without the more active people. The quiet ones may be both somewhat intimidated by the active ones and have learned to "lean" on them. Having to play on their own for a bit might help them build up confidence, while also forcing them to get more active and drive the game on their own. I'm running a campaign right now where the "quiet people" from the last one are my most active players, because the guy who dominated the last campaign I ran with them isn't around anymore. They're now the ones demonstrating how things work to the new people.
Nudging them to contribute more can also help, but be careful; in my experience it works better to offer them opportunities to get involved and to only directly push occasionally, and after they've had some time to warm up. Jumping on them directly at the beginning of the session will often make them freeze. This happens to me sometimes when I'm a player, and it can be very frustrating. Instead, I've had good results with mining their back story for things that will interest and engage them specifically.