OK, I don't have time to answer this as I want to. My background is in psychology, and I fell into role playing games when I turned 10 in 1976. So by the time I was in college, understanding where the term Roleplaying game really came from, I understood the critical nature of immersion, how it is the most important ingredient for game success.
And to be clear, the definition of immersion is to "Immerse oneself into the identity and Role of the part one is playing. To respond, as much as possible, as the person one is playing, not as oneself."
And before getting into the smaller details, I will dive right into the fact that the very system/game one chooses has a huge amount to do with the amount of Immersion.
Metagaming is the opposite of immersion. You use both terms, but I need to make that absolute definition from the beginning. This also means rules that encourage metagaming decrease the immersion in a game and therefore, decrease the main ingredient of a roleplaying game. The mechanics are called "Dissociated Mechanics", a term coined by Justin Alexander. This is very worth reading, because it gets into many of the larger picture issues with players being able to use in-game logic to see the world around them, as opposed to the rules forcing dissociation from in-game logic.
Once the players assume that rules are going to determine the content of an encounter or treasure (based on EL, or whatever) instead of what the environment or history of the area dictate, verisimilitude is lost.
Vreeg's Rules of Setting design are also heavily immersion related. My current campaign is 26 or so years old (started in '83). Building verisimilitude is a huge part of this.
Vreeg's first Rule of Setting Design
Make sure the ruleset you are using
matches the setting and game you want
to play, because the setting and game
WILL eventually match the system.
Corollary to Vreeg's First Rule
The proportion of rules given to a
certain dimension of an RPG partially
dictate what kind of game the rules
will create. If 80% of the rulebook
is written about thieves and the
underworld, the game that is meant
for is thieving. If 80% of the
mechanics are based on combat, the
game will revolve around combat.
- Multiply this by 10 if the reward
system is based in the same area as
the preponderance of rules.
2nd Corollary
Character growth is
the greatest reinforcer. The
synthesis of pride in achievement
with improvement in the character
provides over 50% of the
reinforcement in playing the game.
Rules that involve these factors are
the most powerful in the game.
Vreeg’s Second Rule of Setting Design
Consistency is the
Handmaiden of Immersion and
Verisimilitude. Keep good notes, and
spend a little time after every
creation to ‘connect the dots’. If
you create a foodstuff or drink, make
sure you note whether the bars or inns
the players frequent stock it. Is it made
locally, or is it imported? If so,
where from? If locally made, is it
exported?
Vreeg's Third Rule of Setting Design
The World In Motion is critical
for Immersion, so create 'event
chains' that happen at all levels of
design. The players need to feel like
things will happen with or without
them; they need to feel like they can
affect the outcome, but event-chains
need velocity, not just speed.
Vreeg's Fourth Rule of Setting Design
Create motivated events and
NPCs, this will invariably create
motivated PCs. Things are not just
happening, they happen because they
matter to people (NPCs). There is no
need to overact, just make sure that the
settings and event-chains are
motivated and that the PCs feel
this.
Vreeg's Fifth Rule of Setting Design
The Illusion of Preparedness is critical
for immersion; allowing the players to see
where things are improvised or changed
reminds them to think outside the setting,
removing them forcibly from immersion.
Whenever the players can see the hand of the GM - even when the GM needs to change things in their favor -
it removes them from the immersed position.
(Cole, of the RPGsite, gets credit for the term).
Remember that part of immersion is the lack of feeling walls around and rails under the characters. This means that the players should not feel that there are things that their character cannot do solely because of the rules or the GM's mindset. The job of the GM is to enable roleplay, not to inhibit it.
This also means the GM must be as immersed as the players, or more.
Another big-picture thing that may irk some folk who sell stuff is that published settings can hurt immersion. They don't destroy it; but when the players have a lot of knowledge about a setting that their character would not have, this increases the opportunity to use it, consciously or unconsciously. Similarly, if your setting has its own bestiary that the characters learn as they go along, or at least a lot of homebrew tweaks, the players get used to working with the in-house data and not trusting the published sources.
If you have done all of this larger-scope stuff, the smaller scope stuff becomes easier. As a GM with miles on the tires, I find that playing up the level of knowledge my NPCs might have and do not have helps keep the players in the same mindset. Players key heavily off the way the GM plays their NPCs. They won't do the funny voices or the mannerisms if the GM does not, and if the GM is particularly careful about what their NPCs know and don't know, especially verbally, the players emulate this.
I've had player conflict from time to time. Sometimes it can be handled simply, by talking to them. Be direct - as the GM, you're the leader, and it's your game. "Your fighting makes everyone else not have a good time. I'm the GM, and I'm telling you now to shape up and not bring your beefs into the game."
Look for opportunities to have them play in different games. Most groups I'm in end up having several concurrent games going on on different days or whatever, so people can stay "in the group" but not play with people that bug them. Once, to accommodate differing play styles among the group, I split the campaign and ran two different games on two different days for the two different groups.
Also, sometimes you can address it in game. I had a player who I nearly kicked out, because he would throw fits that were arguably in character but really boiled over to out of character very quickly. (This one time, he was so agitated at the group not taking the thousands of copper pieces from a dead orc tribe that after a screaming fit he just ran into a wraith and suicided.) I ended up giving him a reason to subtly spy on the other players and that sublimated his desire for conflict in a way that didn't have negative group consequences. Perhaps there's ways to sublimate their conflict in the same way. If you disallow table talk, for example, they will have to communicate in character, and arguments in character tend to get you ambushed by bad guys, tossed in dungeons, etc.
In the end, you may have to eject one or both players from your group. Geeks tend to Marty McFly about this too much and not address problems directly, read the Five Geek Social Fallacies for some insight into this (though bands, sports teams, etc. end up having the same problems frequently).
In my current group, we had a player who was just such a total goon that no one enjoyed games that he attended. People tried working with him, but finally we all got together and discussed the problem and ended up voting to disinvite him from the game. "He's making the game less fun" is more than enough justification.
In this case it's harder - is one person really the problem? Or is it both of them, but they're fine if the other isn't there? If they are both a joy to game with except when the other player's there, it's hard to decide which one to punt, but you can just leave it up to them. "Look, we like gaming with both of you, but it's unacceptable and fun-killing when y'all fight. Whoever starts it next time will need to leave." You can try giving people "time outs" short of punting them from the group - "You have to leave this session now because you're fighting." See if that shapes them up. I don't know how much you have invested in these people and how much work is justified to save the group vs. ditching one or more of the members out of hand. It's easy to be an Internet Tough Guy and say "throw them out, then ostracize them from your life!" because it's easy to say that when it's not you in that place. Don't jump to ejection before bothering to talk with friends, but also don't be afraid to pull the trigger once it's clear that's the only thing that will make the game fun again.
Best Answer
If your players are easily frustrated by a few bad dice rolls, that's a problem with your game in general. Bad dice rolls happen. And they will happen a lot. If the laws of the universe don't change in the near future, I would even dare to say they will happen with the same frequency that applies to good dice rolls.
After all, your players don't get frustrated rolling too good, do they? Did they ever leave the table because the rolled 3 criticals in a row totally dominating the encounter? So the real problem probably is, that their only action and therefore their only fun producing aspect of the game is winning the dice roll. That is fine for a board game, but not exactly the goal of roleplaying games.
You can do a lot more in a turn of combat than stand there and hit the enemy. You can move to flank the enemy. You can taunt him (just in character, without any rules involved). You can be creative. You can have a lot of fun with failures. My most memorable moment in roleplaying was when my ninja character failed so miserably sneaking out of a bar that she ended up on the doorsteps with a broken ankle and screaming. Sure, that was a failure, the worst possible combination of dice I ever saw, but it was still fun and a happy memory meeting the people of this group even ten years later. But you need to encourage it and you need to allow it, even if (especially if!) it's not in the rules.
Example: Playing a specialized magic user, our party once met a monster that was completely immune to any of my magic. I could have taken the second (or third) row in combat shooting ordinary arrows at it. I would probably have missed 20 times in a row. It would have been incredibly boring. After 10 misses I would probably have left the table, too. Life is too short to be bored. Instead my character switched into light armour (no proficiency, but who cares), got a dark cape and a dagger and sneaked behind the monster. He failed the sneak roll, he was unable to cast magic due to the armor and he really sucked at hitting it with the dagger. BUT: the GM decided it would turn to me and leave it's back turned to the warrior leaving it open to his attacks, because even the dumbest monster knows that those sneaky dark dagger people hurt the most. I failed every single roll that combat and still contributed and had fun.
Remember, as a GM you need to encourage and allow it. No rule ever said that monsters need to turn to the sneaky git with a dagger. That was good GMing. We had fun. Much more fun than any fireball-damage-dice-rolling-spell could have brought. Build your encounters so that the players feel they can be creative. And allow them to be creative. If the players are creative, then no dice roll is needed to "allow" them to contribute. Contribution is measured in fun, not damage points.
--- Edit: ---
The first part was about what can be changed in the gaming group and playstyle to make failed rolls less frustrating. A commenter remarked that sometimes it has nothing to do with the game or group and that's absolutely right:
Some players simpy cannot lose. They can't. If they lose, they get frustrated. But losing is part of any game. I guess we all know people who are like that. It's also never them. It's the noob team. Or the dice. Or whatever. If the game was lost, somebody must have been a bad player. That you can play a game, be good at it, have fun and still lose doesn't fit in their world view. For them, it's about winning, not about having fun playing. If this is the case with your player, there is little that you can do to change that. For him to be the winner every time, the rest would have to lose. And you can't have that in a group. That's not fair to the others.
Some players are just bad at statistics. They don't have bad luck. They only feel like they had bad luck because they don't know better. Some people need a 18 to hit on a d20 and think missing 3 times in a row is incredibly bad luck. It's not. To the contrary, it would have been pretty lucky to hit just once in three tries. Make sure the players know at least basic statistics and can convert your systems dice rolling to percentages, so they have a number they can grasp.
Some people are building their characters with damage dealing as a priority. They can dish out huge amounts. Many dice. Large numbers. But most systems are quite balanced, so to achieve this, they sacrificed something. Most likely their chance to hit. So they sit there round after round waiting for their one moment of glory where they land a lucky hit to show off their uber damage. That's a decision. They could as well have build a character that attacks three times a round for little damage and in a good system it would have the same end result. As long as they look only at one number (damage) and not at the full picture (damage * chance to hit), they will frustrate themselves every time. Make sure they know that this is their own decision, because they can only change this situation themselves.