Pre-warning - I have little experience as a DM dealing with difficult players, but I'm getting pretty used to dealing with difficult tables in general.
A couple of things we found really handy for our horde of uncooperative players:
Talk it out / Remind him
You said in the comments that:
I have talked to him about it before. He seems understanding until a situation occurs where it basically sets it off
So he may just need a gentle, at table reminder about the discussion you've had. A simple prod in the right direction might be all he needs to cool his head. DnD can get pretty intense at times, and it could just simply slip his mind as he gets into it.
Players really getting into your campaign is what you want as a DM, as long as they're getting into in good spirit.
Is the table okay with this behavior?
Is it just you that's bothered by this? How your table reacts to his behavior could also dictate how you handle it.
During our romps with difficult players, one player refused to cooperate with the DM to the point we almost had to kick him out. But the DM pulled me aside, and asked me to respond to every that player said as in character.
If your brother is heavily into role playing, responding in-kind and keeping it to your characters may help stem the OOC poor behavior.
I.E - If he's complaining about the Gods not lending a hand, a NPC (or other player) could take that at face value. "[Gods Name Here] wouldn't care about a single [town/city/country/]! Our lifes are nothing compared to the grand scheme of things. The immortal plane is beyond our understanding..."
If all else fails, are you guys that right group for him?
You also said that:
He generally seems much kinder to other dms, doesn't have the remarks or throws tantrums to them. (He also is my older brother)
So it could very much be as simple as -Younger brother has to do it my way rawr- or that the tables play-style and campaign just doesn't fit him.
Saying that you play to his level, but then having a level 15 player challenge a lv 20 NPC, says to me that maybe he doesn't want that much of a challenge.
He could be looking for something a bit more cruisy, or maybe he just really really wanted to play pirate captain.
But either way, if you can't find a middle ground with him, maybe have another look to see if he matches your table. It could be as simple as a few small tweaks to the campaign to keep everyone happy.
When he:
Contradicts you
Absolutely unacceptable as a matter of general habit. Talk to him, lay down the position that he needs to accept at least the fact that mid-session, you're the one responsible for adjudicating the rules.
There's some subtlety in how you might want to adjudicate disagreements, but ultimately, mid-session, this is your job.
Says something you disagree with or interferes with you running the game
Disagree with him openly, and see what happens. If he rolls with it perfectly well, there's nothing to do. If he contradicts you, see above. If he doesn't contradict you outright, but this interaction rubs him wrong, discuss this situation with him and phrasings he can use to make this pattern easier. Things such as "Ask the DM if you need to make a check" instead of "make a check".
Says something you were about to say or that you would have wanted to say in retrospect
Roll with it. This isn't a problem. You're getting a free assistant.
Best Answer
I want to lay out three premises to start with:
There is no “right” or “wrong” here. You should throw out any preconception you have about what is “right” or “wrong” here; no one is either of those things. Laying blame, meting out punishment, etc. is not appropriate and will not help. Cajoling the player with rewards isn’t really quite right either. I strongly suggest you stop these things immediately. Were it me, really no matter what I was doing, having the other PCs laugh at me really would be a very strong incentive to leave the game. So I strongly suggest that these measures are not the right approach, and you should stop trying them.
There is an issue here, even if no one is “right” or “wrong.” Just because no one here is “right” and no one here is “wrong” doesn’t mean there’s no problem: there is. Brief disengagement from the game following a string of bad luck leaving you feeling unable to contribute is not terribly unusual, nor is it necessarily terrible for the game in and of itself, but only if this is brief and infrequent. You are suggesting that it is both frequent and quite extended. That suggests a player who is not having fun, and that’s a problem, even if it is no one’s “fault.” Fun is why we play, after all.
There may not be a solution. The things necessary for this player to have fun, to avoid these bouts of disengagement, may not be things that would make the game fun for you or the other players. This is still not a question of “right” or “wrong”—but it does mean that the issue before the table is intractable. Agreeing to part ways is a valid resolution; an undesirable one, but still better than someone continuing to spend their time doing things they don’t enjoy, or getting into a fight about it.
So the perspective to take here is to try to determine ways for this player to have more consistent fun, to avoid these bouts of disengagement, without tarnishing the fun for yourself or others, and so allow the game to continue without losing anyone or having someone there who isn’t having fun.
The first thing to do is to talk to the player, probably privately, definitely out-of-character. You need to determine a number of things, but most important of which is 1. is this player actually enjoying the game overall? and 2. how that player feels about the disengagement; do they feel it’s an appropriate reaction to frustration and are fine with it, or are they expressing more severe dissatisfaction this way. You should also very seriously consider apologizing for the ham-handed ways you’ve tried to address this in the past.
The second thing to do is to try to address the underlying problem: bad rolls are disheartening to a lot of players. It’s perfectly normal to be disappointed by them in games where you are seeking to achieve some goal through your character, and a failed roll means you are not making progress toward that goal.
Other game systems can avoid this entirely by focusing more on producing a story than achieving a goal, often aided by more “interesting failure” than merely “you didn’t accomplish the thing you were trying to.”
Which is what brings me to my suggestion: player engagement improves when players get to do things. You can offer them alternative options that don’t require rolls, or require lower numbers. You can even offer them effects on missing: you are welcome to say things like “the orc parries your blow, but you could follow the blow through into the rope holding up the chandelier if you wanted,” for example. Being creative here is good—getting your players to be creative here is better. Obviously whatever happens on a miss should—usually—be less than what happens on a hit, but offering circumstantial lesser effects can really bring a struggle to life.