I too have a "rules lawyer" and power gaming player in my group, who also wanted a homebrew class/subclass to fit his character and tends to create his characters as efficient as it gets. So most of these problems I have also dealt with.

He made his own PC race without my permission, which can shapeshift and has whatever racial bonuses he wants.

(In my case, the player wanted to play a cleric but had an idea which was not represented in the book, so he wanted to make his own. He created a first draft of how he thought the class should look like, then I adjusted it, asked reddit (who were rude but helpful) and then we put his character into play thematically the way he wanted him to be, but mechanically adjusted so he was not stronger than the other cleric options.)

The relevant part is that if we notice that the balance is not right we can and will adjust his character between sessions. (This also includes improving.)

I would recommend you to start "underpowered" and later buff his abilities if needed, this feels way less depressing for everybody than getting nerfed all the time.

An example to adjust his "shapeshifter":

• He can only shapeshift into humanoid races (the creature type, so no beasts or something else that is restricted to Druids and the Shapechange spell)
• Limit his abilities in terms of time and usages. Like "only once per short rest" and "the shapechanger is able to hold his changed form for 1 hour until he returns to default". (Look up Doppelganger in the MM for his default form and create racism. No one would like a doppelganger in his store.)

Keep in mind that this shape change is not only "I use any race benefit I want", but also a very effective and cheap way to gain the effect of the Alter Self spell without using a spell slot. He could steal something in a different form and nobody would guess it was him — unless they somehow know he is a doppelganger/shapechanger, and then suddenly he is always being accused. (Maybe don't be that harsh.)

If you are not down for homebrew at all, then tell him upfront and do not allow him to use any source which is not official. You have to be strict then. Tell him that the other players don't have a custom race, why should he?

He always wants to kill a specific PC, [...]

(For quite some time PvP was a huge topic in my playing group. Said player asked me questions like, would he be able to cast Contagion on another PC without him noticing. I do not want PvP in my group, even if all players are down for it. It is not the game I want to play, and so I told him that if he decides to engage in PvP he will have to create a new character, because either the other PCs or fate (me) will make sure his old character will no longer be part of the game. He created a new character fitting better into the group, and the old one simply left the group with the excuse that he no longer sees any reasons to travel with them.)

From your question, I recognize you do not want any PvP either. This is something one should clearly define in a so-called "session 0". Do you want it to be a PvP game? Do the players want it? If not, say so clearly, and do not allow characters which do not fit into the group or are created solely to kill other PCs.

My approach is the following

• If you introduce a new character, you have to get the rest of the group to accept him into the party. I do not use divine intervention and say "he is in your group now".
• The background of one's character is created by the player who plays it and only him (in collaboration with the DM). That way there is no way a player can say "I am an assassin hired to kill the other PC", because he is not allowed to tamper with the other PCs' background and add someone who wants to see him dead.

This player is a huge rules lawyer. [...]

When improvising rules, I think the best thing to do is that you write down your ruling and after the session look if there is an official rule which fills this gap. If yes, take the official one, if not or if you enjoy your own rule more, then complete your homebrew rule in a rule text. This makes the rule less arbitrary and defines clear limits. You already did that by creating the trait.

He pouts when it isn't his turn to do something [...]

I see three approaches to that problem.

1. This is probably the worst one. Ignore him. Show him that it is not always about him.
2. Speed up combat by encouraging players to preplan their turns. In that way, none of them have to wait very long until it's their turn again, and if it is not their turn they have to plan and adjust their plans to whatever happens on the battlefield.
3. Allow strategic "out of character" planning. Maybe it will cheer him up if he can talk about other players' options (if he does not disturb them by doing so).

[...] and he always wants to be the centre of attention.

I suggest creating situations that other PCs are simply better in, so it is their time to shine.

• A Druid is better in a forest/survival situation.
• A Barbarian is better at destroying obstacles.

Create multiple and/or fake spotlights:

• If he is the one always walking in front of the group, let him be the first one who walks into traps or ambushes.
• If he is talking to an NPC, he is distracted and does not recognize a thief stealing his bag of holding, while his party members are not distracted and see it.
• In some situations, you have to switch the spotlight, so there is more than one 'center' at the same time. The guy in a discussion gains information while the other PCs who do not want to listen to him witness something that can help them. I am talking about things that happen in parallel. He cannot be everywhere at the same time.

He once threatened me with a Nerf gun because I wasn't going to let him keep a roll he made prematurely.

I do not know how serious he was at that moment, but that would be a situation where I would have thrown him out immediately. The game is about fun.

Also, if he is such a rules lawyer, he should know that you only roll the dice if the DM tells you to, and if it is your turn and you announced what you are rolling for. Even without the Nerf gun ridiculousness, I would have told him that if he does not roll "again" he chooses to fail the check.

His characters have been too powerful and prideful and he always is trying to max out the system, so he can be the most powerful character. [...]

(I know this too well. The first two or three characters of the player I am referring to were realy powerful. He since learned that this maxing is not necessary with me as DM, and is currently playing a very squishy character that, because his character does not understand how the world really works, says stupid things in dialogs and chooses non-optimal ways to go, because they are more fun. I do not know how I got him to enjoy the roleplaying more than the "power of his character". I guess it was because the other players did not participate in his competition.)

When he sticks to the rules there is nothing you can do. (Well, there is, but I wouldn't suggest you do that.)

If he uses homebrew, see question 1.

This is a pretty common problem, I think it is a different play style. He enjoys being an optimized powerful hero and probably likes the combat system the most, while you and the other players want to have a lightweight, more roleplay-y game.

In that case I would suggest:

• Give them non-combat options to solve problems

• Increase the number and the relevance of social encounters

• Use/create enemies/encounters which his character is not built for and other characters are more useful/helpful against, like:

  • Is he a melee fighter? Flying enemies, ranged enemies which are faster than him and/or use the environment to keep him out of reach, slowing spells.
  • Is he a ranged fighter? Fast enemies which engage him in melee combat, strong wind or other conditions (spells?) which protect against ranged attacks.
  • Is he a sorcerer? Enemies with high wisdom saving throws if he uses those spells, enemies with antimagic (Beholder, counterspell, dispel magic, magic resistance).

Best would be mixed encounters where the group has to play as a team and are depending on one another to stick to what they are best at. Maybe tune up the encounter difficulty so he learns that he cannot fight alone and needs help from the others. Or maybe tune down the encounter difficulty, so he learns that maxing is not only just not required, but also unfun because it is to easy.

I did the latter, because I balanced around the other characters.

Summary

It is your game and the typical answer on problem-players would be that the game is not only about him, and if he is not enjoying the game and/or disturbs the fun of you and/or your group, he might be better off in a different game.

And this is not a bad thing. Different people enjoy different play styles. I get the feeling that he is more the guy for an old school, encounter heavy, deadly dungeon crawl.

For your future games

I would suggest a "session 0" where the group sits together and talks about what kind of game they want to play. An official adventure? A homebrewed one? A sandbox-y game?

In what world are we? Forgotten Realms? Eberron? Something homebrewed?

Are we playing good, neutral or evil characters? Do we want to be heroes, villains, or just selfish dudes who happen to be pulled into something world changing?

Is PvP okay? How are our characters connected? A random event? Do we belong to the same group? (I would suggest having them be part of a group or know each other for about 3 years already, the first would also make an easy hook for new characters joining the party.)

Do we primarily deal with combat? Is it a social campaign with intrigues, etc.? How roleplay-y are we going to get?

And so on...

Also: Clear up the homebrewed rules you want to use. Since you already play a game right now, you have your house rules at the ready and can explain them right at the beginning for everyone.

It is very good storytelling.

One of my top tips for DMs is, if you can shove player's past into the mix and make it look flawlessly pre-meditated, the players will adore you. It is always good to have the player's backstories come haunt them enough to push them into action but little enough as to not make an entire campaign centered around two of these characters.

But that doesn't mean that the spotlight can't shine over one or two players every now and then. That and, hey! It lets them also integrate with each other, and a party that's integrated is a party that will participate with the story much better and runs less risk of splitting up.