OK, I don't have time to answer this as I want to. My background is in psychology, and I fell into role playing games when I turned 10 in 1976. So by the time I was in college, understanding where the term Roleplaying game really came from, I understood the critical nature of immersion, how it is the most important ingredient for game success.
And to be clear, the definition of immersion is to "Immerse oneself into the identity and Role of the part one is playing. To respond, as much as possible, as the person one is playing, not as oneself."
And before getting into the smaller details, I will dive right into the fact that the very system/game one chooses has a huge amount to do with the amount of Immersion.
Metagaming is the opposite of immersion. You use both terms, but I need to make that absolute definition from the beginning. This also means rules that encourage metagaming decrease the immersion in a game and therefore, decrease the main ingredient of a roleplaying game. The mechanics are called "Dissociated Mechanics", a term coined by Justin Alexander. This is very worth reading, because it gets into many of the larger picture issues with players being able to use in-game logic to see the world around them, as opposed to the rules forcing dissociation from in-game logic.
Once the players assume that rules are going to determine the content of an encounter or treasure (based on EL, or whatever) instead of what the environment or history of the area dictate, verisimilitude is lost.
Vreeg's Rules of Setting design are also heavily immersion related. My current campaign is 26 or so years old (started in '83). Building verisimilitude is a huge part of this.
Vreeg's first Rule of Setting Design
Make sure the ruleset you are using
matches the setting and game you want
to play, because the setting and game
WILL eventually match the system.
Corollary to Vreeg's First Rule
The proportion of rules given to a
certain dimension of an RPG partially
dictate what kind of game the rules
will create. If 80% of the rulebook
is written about thieves and the
underworld, the game that is meant
for is thieving. If 80% of the
mechanics are based on combat, the
game will revolve around combat.
- Multiply this by 10 if the reward
system is based in the same area as
the preponderance of rules.
2nd Corollary
Character growth is
the greatest reinforcer. The
synthesis of pride in achievement
with improvement in the character
provides over 50% of the
reinforcement in playing the game.
Rules that involve these factors are
the most powerful in the game.
Vreeg’s Second Rule of Setting Design
Consistency is the
Handmaiden of Immersion and
Verisimilitude. Keep good notes, and
spend a little time after every
creation to ‘connect the dots’. If
you create a foodstuff or drink, make
sure you note whether the bars or inns
the players frequent stock it. Is it made
locally, or is it imported? If so,
where from? If locally made, is it
exported?
Vreeg's Third Rule of Setting Design
The World In Motion is critical
for Immersion, so create 'event
chains' that happen at all levels of
design. The players need to feel like
things will happen with or without
them; they need to feel like they can
affect the outcome, but event-chains
need velocity, not just speed.
Vreeg's Fourth Rule of Setting Design
Create motivated events and
NPCs, this will invariably create
motivated PCs. Things are not just
happening, they happen because they
matter to people (NPCs). There is no
need to overact, just make sure that the
settings and event-chains are
motivated and that the PCs feel
this.
Vreeg's Fifth Rule of Setting Design
The Illusion of Preparedness is critical
for immersion; allowing the players to see
where things are improvised or changed
reminds them to think outside the setting,
removing them forcibly from immersion.
Whenever the players can see the hand of the GM - even when the GM needs to change things in their favor -
it removes them from the immersed position.
(Cole, of the RPGsite, gets credit for the term).
Remember that part of immersion is the lack of feeling walls around and rails under the characters. This means that the players should not feel that there are things that their character cannot do solely because of the rules or the GM's mindset. The job of the GM is to enable roleplay, not to inhibit it.
This also means the GM must be as immersed as the players, or more.
Another big-picture thing that may irk some folk who sell stuff is that published settings can hurt immersion. They don't destroy it; but when the players have a lot of knowledge about a setting that their character would not have, this increases the opportunity to use it, consciously or unconsciously. Similarly, if your setting has its own bestiary that the characters learn as they go along, or at least a lot of homebrew tweaks, the players get used to working with the in-house data and not trusting the published sources.
If you have done all of this larger-scope stuff, the smaller scope stuff becomes easier. As a GM with miles on the tires, I find that playing up the level of knowledge my NPCs might have and do not have helps keep the players in the same mindset. Players key heavily off the way the GM plays their NPCs. They won't do the funny voices or the mannerisms if the GM does not, and if the GM is particularly careful about what their NPCs know and don't know, especially verbally, the players emulate this.
Pre-warning - I have little experience as a DM dealing with difficult players, but I'm getting pretty used to dealing with difficult tables in general.
A couple of things we found really handy for our horde of uncooperative players:
Talk it out / Remind him
You said in the comments that:
I have talked to him about it before. He seems understanding until a situation occurs where it basically sets it off
So he may just need a gentle, at table reminder about the discussion you've had. A simple prod in the right direction might be all he needs to cool his head. DnD can get pretty intense at times, and it could just simply slip his mind as he gets into it.
Players really getting into your campaign is what you want as a DM, as long as they're getting into in good spirit.
Is the table okay with this behavior?
Is it just you that's bothered by this? How your table reacts to his behavior could also dictate how you handle it.
During our romps with difficult players, one player refused to cooperate with the DM to the point we almost had to kick him out. But the DM pulled me aside, and asked me to respond to every that player said as in character.
If your brother is heavily into role playing, responding in-kind and keeping it to your characters may help stem the OOC poor behavior.
I.E - If he's complaining about the Gods not lending a hand, a NPC (or other player) could take that at face value. "[Gods Name Here] wouldn't care about a single [town/city/country/]! Our lifes are nothing compared to the grand scheme of things. The immortal plane is beyond our understanding..."
If all else fails, are you guys that right group for him?
You also said that:
He generally seems much kinder to other dms, doesn't have the remarks or throws tantrums to them. (He also is my older brother)
So it could very much be as simple as -Younger brother has to do it my way rawr- or that the tables play-style and campaign just doesn't fit him.
Saying that you play to his level, but then having a level 15 player challenge a lv 20 NPC, says to me that maybe he doesn't want that much of a challenge.
He could be looking for something a bit more cruisy, or maybe he just really really wanted to play pirate captain.
But either way, if you can't find a middle ground with him, maybe have another look to see if he matches your table. It could be as simple as a few small tweaks to the campaign to keep everyone happy.
Best Answer
Use visual reminders
Sometimes, a player/DM like this only needs a visual cue that you've discussed in advance to remind them that they're engaging in behaviour you've previously talked about. Since this person is open and receptive, but keeps falling into habitual metagaming, start using a visual cue to remind them.
I recommend a pack of playing cards. Just tell him/her that the cards are meant to serve as a reminder to try to pull back the metagaming. Every time they start, just hand them a card. If the person is as receptive as you said, they'll check themselves immediately.
After time, this will no longer be necessary. You'll have effectively trained them out of a bad habit. Be sure to reward said behaviour with some cold beer to reinforce the change. :)
This advice comes from a lot of experience dealing with players who used to DM and actually falling into this same habit myself. Visual cues can be very subtle, and very powerful.