I too have a "rules lawyer" and power gaming player in my group, who also wanted a homebrew class/subclass to fit his character and tends to create his characters as efficient as it gets. So most of these problems I have also dealt with.
He made his own PC race without my permission, which can shapeshift and has whatever racial bonuses he wants.
(In my case, the player wanted to play a cleric but had an idea which was not represented in the book, so he wanted to make his own. He created a first draft of how he thought the class should look like, then I adjusted it, asked reddit (who were rude but helpful) and then we put his character into play thematically the way he wanted him to be, but mechanically adjusted so he was not stronger than the other cleric options.)
The relevant part is that if we notice that the balance is not right we can and will adjust his character between sessions. (This also includes improving.)
I would recommend you to start "underpowered" and later buff his abilities if needed, this feels way less depressing for everybody than getting nerfed all the time.
An example to adjust his "shapeshifter":
- He can only shapeshift into humanoid races (the creature type, so no beasts or something else that is restricted to Druids and the Shapechange spell)
- Limit his abilities in terms of time and usages. Like "only once per short rest" and "the shapechanger is able to hold his changed form for 1 hour until he returns to default". (Look up Doppelganger in the MM for his default form and create racism. No one would like a doppelganger in his store.)
Keep in mind that this shape change is not only "I use any race benefit I want", but also a very effective and cheap way to gain the effect of the Alter Self spell without using a spell slot. He could steal something in a different form and nobody would guess it was him — unless they somehow know he is a doppelganger/shapechanger, and then suddenly he is always being accused. (Maybe don't be that harsh.)
If you are not down for homebrew at all, then tell him upfront and do not allow him to use any source which is not official. You have to be strict then. Tell him that the other players don't have a custom race, why should he?
He always wants to kill a specific PC, [...]
(For quite some time PvP was a huge topic in my playing group. Said player asked me questions like, would he be able to cast Contagion on another PC without him noticing. I do not want PvP in my group, even if all players are down for it. It is not the game I want to play, and so I told him that if he decides to engage in PvP he will have to create a new character, because either the other PCs or fate (me) will make sure his old character will no longer be part of the game. He created a new character fitting better into the group, and the old one simply left the group with the excuse that he no longer sees any reasons to travel with them.)
From your question, I recognize you do not want any PvP either. This is something one should clearly define in a so-called "session 0". Do you want it to be a PvP game? Do the players want it? If not, say so clearly, and do not allow characters which do not fit into the group or are created solely to kill other PCs.
My approach is the following
- If you introduce a new character, you have to get the rest of the group to accept him into the party. I do not use divine intervention and say "he is in your group now".
- The background of one's character is created by the player who plays it and only him (in collaboration with the DM). That way there is no way a player can say "I am an assassin hired to kill the other PC", because he is not allowed to tamper with the other PCs' background and add someone who wants to see him dead.
This player is a huge rules lawyer. [...]
When improvising rules, I think the best thing to do is that you write down your ruling and after the session look if there is an official rule which fills this gap. If yes, take the official one, if not or if you enjoy your own rule more, then complete your homebrew rule in a rule text. This makes the rule less arbitrary and defines clear limits. You already did that by creating the trait.
He pouts when it isn't his turn to do something [...]
I see three approaches to that problem.
- This is probably the worst one. Ignore him. Show him that it is not always about him.
- Speed up combat by encouraging players to preplan their turns. In that way, none of them have to wait very long until it's their turn again, and if it is not their turn they have to plan and adjust their plans to whatever happens on the battlefield.
- Allow strategic "out of character" planning. Maybe it will cheer him up if he can talk about other players' options (if he does not disturb them by doing so).
[...] and he always wants to be the centre of attention.
I suggest creating situations that other PCs are simply better in, so it is their time to shine.
- A Druid is better in a forest/survival situation.
- A Barbarian is better at destroying obstacles.
Create multiple and/or fake spotlights:
- If he is the one always walking in front of the group, let him be the first one who walks into traps or ambushes.
- If he is talking to an NPC, he is distracted and does not recognize a thief stealing his bag of holding, while his party members are not distracted and see it.
- In some situations, you have to switch the spotlight, so there is more than one 'center' at the same time. The guy in a discussion gains information while the other PCs who do not want to listen to him witness something that can help them. I am talking about things that happen in parallel. He cannot be everywhere at the same time.
He once threatened me with a Nerf gun because I wasn't going to let him keep a roll he made prematurely.
I do not know how serious he was at that moment, but that would be a situation where I would have thrown him out immediately. The game is about fun.
Also, if he is such a rules lawyer, he should know that you only roll the dice if the DM tells you to, and if it is your turn and you announced what you are rolling for. Even without the Nerf gun ridiculousness, I would have told him that if he does not roll "again" he chooses to fail the check.
His characters have been too powerful and prideful and he always is trying to max out the system, so he can be the most powerful character. [...]
(I know this too well. The first two or three characters of the player I am referring to were realy powerful. He since learned that this maxing is not necessary with me as DM, and is currently playing a very squishy character that, because his character does not understand how the world really works, says stupid things in dialogs and chooses non-optimal ways to go, because they are more fun. I do not know how I got him to enjoy the roleplaying more than the "power of his character". I guess it was because the other players did not participate in his competition.)
When he sticks to the rules there is nothing you can do. (Well, there is, but I wouldn't suggest you do that.)
If he uses homebrew, see question 1.
This is a pretty common problem, I think it is a different play style. He enjoys being an optimized powerful hero and probably likes the combat system the most, while you and the other players want to have a lightweight, more roleplay-y game.
In that case I would suggest:
Give them non-combat options to solve problems
Increase the number and the relevance of social encounters
Use/create enemies/encounters which his character is not built for and other characters are more useful/helpful against, like:
- Is he a melee fighter? Flying enemies, ranged enemies which are faster than him and/or use the environment to keep him out of reach, slowing spells.
- Is he a ranged fighter? Fast enemies which engage him in melee combat, strong wind or other conditions (spells?) which protect against ranged attacks.
- Is he a sorcerer? Enemies with high wisdom saving throws if he uses those spells, enemies with antimagic (Beholder, counterspell, dispel magic, magic resistance).
Best would be mixed encounters where the group has to play as a team and are depending on one another to stick to what they are best at. Maybe tune up the encounter difficulty so he learns that he cannot fight alone and needs help from the others. Or maybe tune down the encounter difficulty, so he learns that maxing is not only just not required, but also unfun because it is to easy.
I did the latter, because I balanced around the other characters.
Summary
It is your game and the typical answer on problem-players would be that the game is not only about him, and if he is not enjoying the game and/or disturbs the fun of you and/or your group, he might be better off in a different game.
And this is not a bad thing. Different people enjoy different play styles. I get the feeling that he is more the guy for an old school, encounter heavy, deadly dungeon crawl.
For your future games
I would suggest a "session 0" where the group sits together and talks about what kind of game they want to play. An official adventure? A homebrewed one? A sandbox-y game?
In what world are we? Forgotten Realms? Eberron? Something homebrewed?
Are we playing good, neutral or evil characters? Do we want to be heroes, villains, or just selfish dudes who happen to be pulled into something world changing?
Is PvP okay? How are our characters connected? A random event? Do we belong to the same group? (I would suggest having them be part of a group or know each other for about 3 years already, the first would also make an easy hook for new characters joining the party.)
Do we primarily deal with combat? Is it a social campaign with intrigues, etc.? How roleplay-y are we going to get?
And so on...
Also:
Clear up the homebrewed rules you want to use. Since you already play a game right now, you have your house rules at the ready and can explain them right at the beginning for everyone.
Make their knowledge uncertain
Tell them: "This my game and my world. I might change some things that you might have read in the books or the module."
Now the player cannot ever be sure that their background knowledge is accurate. It might be and there is a high chance it is, but not 100%. Almost like they just heard a few (possibly tall) tales or myths about the monsters and dungeons. Which is a completely reasonable assumption about a character. Their player knowledge just became character knowledge.
Also note that I did not say to change anything, just to say so. Actually changing anything is not required for this to work, but you might want to at least throw some minor changes in just so they or you can point at them and say "Aha! A difference!".
About the insight roll example
This is not really a metagaming issue, but a problem with communication. You and the player do not have the same understanding of what a nat1 means and/or how certain the evidence gained later is. Do not be afraid to clearly state what you mean. Descriptive and immersive narration has its limits and sometimes you have take a step back to avoid misunderstandings. However, even if you do that, there might be issues. You cannot completely stop a player from just running with a misunderstood idea. Agency over their character gives them the power and right to do that. If they choose not to listen to your descriptions, you can only try to compensate. I would recommend making the same evidence available to other PC-s so they can confront the character in the game.
If the issue persists or reappears, then it is not an in-game problem, but an out-of-game one. As such it should be solved as one, by talking to the player about it.
Best Answer
Definition
I'm working to the Wikipeadia definition:
Disclaimer
Metagaming is not ipso facto a bad thing and it happens in every game to a greater or lesser extent. The fact that you ask the question inherently assumes that metagaming is bad (otherwise there is no point to the question): I raise this to explicitly call out your not necessarily correct assumption.
Each table must decide for themselves what they consider is unacceptable metagaming as opposed to what they will permit. Of course, if it's allowed in your game then, by the definition above, it stops being metagaming because the game now accommodates it.
For example, a professional football coach choosing his lineup and strategy for Saturday's match based (partly) on who the opposition is, is metagaming: the strategy is informed by things that are outside the rules of the game of football. However, when considered within the completion as a whole it's not metagaming and any coach who didn't do it would not be a coach for long.
The answer
The initiative order and the effects of spells:
The only aspect of metagaming that applies here is one of verisimilitude - you seem to think that initiative order is something known to the players but not the characters. If so, then your problem goes deeper than deciding what spell to use based on that order - your mind should be rebelling at the fact that people in combat take turns in the first place!
There are two ways of reconciling this:
It's actually a fundamental feature of the fantasy world that includes dungeons and dragons, magic and monsters. If the PCs were playing us they would be equally discombobulated that combat here happens in a wildly simultaneous way.
The abstraction that we use is hiding the fact that trained people in combat actually coordinate their actions. The spell grants advantage partly because of the magic, partly because the spell caster picked the opportune time to cast it and partly because the attacker knew it was coming because she and the spell caster are part of a tight-knit fighting machine and they practice this stuff off-camera.
In either case, there is a narrative analogue to the mechanical resolution which allows whatever small metagaming there is here (if any) to be safely ignored.
As a final point, without knowledge of the mechanics of the game the spell would be rather useless. This is true of a large number of spells and other effects: the spell Guidance, Bardic Inspiration and the Lucky feat all spring to mind.