I think there's a misalignment of the game the two sides want to play. The wonderful thing about D&D is that there's no right way to play it. You could spend an entire campaign cutting deals and playing a trader, provided the table can scrape together house rules sufficient to make that work. Alternately, you could play a dungeon crawler and effectively be playing a bunch of sociopaths that massacre their way through drow in the Underdark so that they can steal their lucky charms. Most groups run a more in the middle path where they work with a DM's provided NPCs to get their marching orders to go kill the drow and steal their lucky charms for the NPC. There's of course many other ways to play as well.
Regarding your specific problem, it appears to me that you and your players are not seeking the same game, and one or both sides isn't acknowledging that this is a cooperative game between DM and the players. Both sides need to be interested in the same play style for things to work at the table. Unfortunately, it appears that a discussion didn't occur early on to handle this matter and it appears you let your frustration on the matter explode at the table, and now everyone's cowed into submission for fear of angering you again and possibly the whole game being scrapped as a result.
The fact that the players have shown up and so dramatically reversed course is, in my opinion, a good thing for you. This means that they want to play, but they don't want it to be a burden for you. So I would recommend the following prior to your next session:
- Apologize for your behavior at the outset. You may have been right, but how you presented it may not have been. It brings you back down to their level and helps remind everybody that we're all friends here. D&D is not a game worth losing friends over.
- Discuss with the group how they envision the game should be and compare it to how you view it. Be advised that they may feel compelled to state that they want to be super diplomats, so ask if they had more fun playing murder hobos. It is extremely important to acknowledge that neither way is 'right'. We're playing make-believe, so it's a lot less necessary to worry how the Orc chieftain feels after the session's over.
- Be sure to discuss how you want the game to be, but be sure to provide a bit of "why" on that. You are the DM and the arbiter of the rules, but if you've no players then you're god of a pretty barren wasteland.
If you spend the entire session having this discussion and get no playtime in, that's fine. This is a very helpful discussion to have at times.
Personally, I occasionally have the opposite problem where I blow up a the table because I don't like the way a rule gets adjudicated. I've gotten better at it, but I do need to check myself and apologize when I get out of hand. I think folks would prefer it didn't happen at all, but when it does a lot can be accomplished with a sincere apology and discussion. This has gotten easier as I've gotten older (I'm 33) compared to when I was 20 something and knew everything.
Overall, your position is untenable
Most of your argument hinges on the role of the DM in the group - you specifically cite p.5-6 of the PHB. However, this defines the role of the DM within the group: at issue is the composition of the group as a whole and the DM has no special role or authority in that.
The formation and ongoing evolution of a social group is the prerogative of the group members (and prospective members) individually and collectively not the rules of the activity they intend to engage in. For example, the rules of a football association (written and unwritten) are not the rules of football - for one of the members to assert authority because they act as the referee on the field is a category error. Similarly the social rules of a D&D group are not the rules of D&D.
In particular
Inclusion of new members of the campaign are group decisions.
Indubitably - see above.
Introducing new characters towards the end of the campaign disrupts the narrative.
It all depends on how this is handled: maybe yes, maybe no. There is probably a great question on its own in this one statement - please ask it but please read good subjective/bad subjective first.
By making an executive decision under DM purview from the get go, I am undermining their perspectives and feelings by "laying down the law" and not listening to their side. (This might be true.)
First, the decision is not within your purview as explained above. Second, there's no "maybe" about it - this is absolutely what you are doing.
The player is part of the pre-existing group and is not a new member as this adventure is an extension of main campaign.
This is the best reason given for including the player but is not a compelling reason on its own.
While players are encouraged to add to the narrative, what is and is not narrative appropriate falls under the domain of the DM as stated on pages 5 & 6 of the 5e Players Handbook.
Really? Go read those pages again because they say no such thing. Those pages describe a dialog between the player's and the DM from which a narrative emerges. The DM controls the environment the players control the PCs - you need both to have a narrative.
Notwithstanding - see above.
Additionally, I have interesting narrative plans involved with his character, that several players and DM's outside of this group approve of.
The first part of this is the second best argument for inclusion. The second part is an irrelevancy - people outside the group don't matter (including me).
As the DM, I serve as referee. ...
See above.
As a DM I have a prerogative to enjoy myself during these sessions else I lose interest in the game and it dies.
Absolutely. However, all the other players have this right too and where your rights and their rights are in conflict, that conflict has to be resolved. "My way or the highway" is a method of dispute resolution - it is unlikely to be an optimal method.
How to fix it
Apologize - always a good first step in making things better. Even if you disagree with everything I say and maintain that your position was 100% correct and above reproach I am sure you are sorry about the way things turned out. You wouldn't be asking this question otherwise.
Reset and restart - decide if you want to (or can) go back to the status quo ante, or continue from here, or euthenise the campaign.
Discuss - listen to their arguments and understand their position. Explain yours. Make arguments that will change people's minds, not ones that entrench their opinions against you. Divisive arguments that might work in a courtroom are unlikely to work in a negotiation - after all you are rarely looking for an ongoing relationship in a courtroom. Know what your power and influence in the group actually are, not what you think they are.
Resolve - make a group decision that the group can live with even if individual members (including you) are not totally happy with.
Learn - always learn.
Best Answer
Switch to Paranoia.
Paranoia is an RPG where the players are expected to backstab (and
occasionallyfrequently frontstab) each other. If you die, just pop out one of your backup clones. This way the players can play the game they want to play without hurting (and in fact improving!) the overall session.Then talk to them.
If you want to run a serious game, the best thing to do is sit everyone down and make it clear that you want to run a serious game. The players might be okay with it, but never realized that you really wanted to do this. Alernatively, they might not be okay with it, in which case forcing them to cooperate is just going to make everybody involved really unhappy. That's when you need to find another group.