So I am a new to D&D but I have researched hours upon hours about it, to the point I know more than probably all my players. They are 7 for now (third level bard, forge cleric, ranger, arcane archer fighter, wizard, barbarian and sorcerer), 2 of them experienced, the rest new to the game. I decided to go as a DM because I felt creative, and it is hard to find a DM.
First session went great, it went exactly how I wanted to, despite having alternatives. The players enjoyed it a ton and we had great fun.
Second session however.. Some players got angry at each other for picking "bad" classes and not balancing their stats (this player got in the game last minute, I was annoyed but found a way to fit him in, but he did not prepare his character as detailed as the others, so he got angry he had a simple ranger. He picked his character, mind you), for not reading their spell notes and checking them in their turn while a whole fight happened at the same time, and then because they did not manage to save an NPC that got kidnapped. She got kidnapped by a bunch of sahuagin, koalinth and a merrow, with their sharks lurking in the water. I had planned for her to get kidnapped, as she would be turned into the first main boss later on when they gave chase to save her, depending on their actions. But they got pissed it didn't go their way, and "wasted resources" (I mean, it was a big battle, of course you would use your spells, that's the point. They were 7, 8 if you count the NPC, against 16, killed 13 of them, the remaining three escaped almost dead with the NPC last second).
What am I supposed to do as a DM to calm them down? I tried to but they wouldn't stop whining, and then they got angry for the simplest of things, like how the Merrow underwater could see on his right the shark that got attacked (he was charmed, but that too was a mistake on my part, he is a monstrosity so he should not be effected from calm emotions, yet I gave it to them and they still whined) and he broke free of the spell, while they can see detailed the whole map and adjust their actions. Or whining when a sahuagin grabs ahold of the barbarians weapon trying to wrestle with him, and he can't throw his javelin in his turn, but instead needs to break free of the sahuagin.
I lost my patience after some time, being calm and them doubting everything, when they thought 30 minutes about their characters (max), and I have thought about the whole campaign and designed it for days upon days, and thought about alternatives, too. I don't have a problem with them suggesting or reminding me of things, it happened many times in these two sessions and I accepted the suggestions, but when they angrily whine and complain about something, I cannot allow it for their sake.
So I ask the veterans help. What could I do to make this a better experience for all of us? What mistakes did I make and what did they make? How can I be balanced but at the same time make the players feel heroic? And how do I make them understand that not everything can go their way, it's up to the dice and of course if the DM allows it. The battle was too big maybe: should I have used stronger but fewer enemies? The battle took around 2.5-3 hours, with a break, they used a lot of time at talking with each other too: so is this a good time for such big battle, or too much?
Best Answer
There are a lot of things going on here, and I think that they may be amplifying each other so that the sum is worse than its parts. I'm going to list off a bunch of things that seem like problems to me, but I want you to know I'm not just doing this to beat up on you. (Especially since it sounds like you're not just new to GMing, but new to D&D in general. Some of this stuff just has to be experienced before it really sinks in.)
Also, I have to list them before I can give out advice.
Some Problems
Big Groups Are Hard To GM For
They really are. And my experience is that the complexity does not scale linearly, meaning, the jump from four to five players is one thing, but the jump from five to six is bigger, and the jump from six to seven is bigger still. There are reasons for this, but knowing why is less important than knowing that it is.
Every GM is going to have their sweet spot for what they prefer (for me, five is a good number) but when in doubt modern gaming systems are usually designed for a certain range of players, and 5e tends to work well with about 4 or 5 plus a GM.
Starting At Higher Than Level One
This isn't always a bad thing-- I tend to like to start at mildly higher levels in different systems because first level doesn't always feel heroic. In some systems, it isn't meant to.
But there are severe drawbacks for both new GMs and new players. Third level characters, especially spellcasters, have a lot more options in play than first level ones. I can virtually guarantee that this is where some of...
...was coming from. It shows up in a lot of ways, too: There's more abilities to remember. There's more abilities to internalize and understand on an intuitive level. (The first is just remembering what things you can do; the second is having a quick feel for how well something will work.) There's a lot more for you, the GM to remember in combat. And with seven players, there's a lot of time as you go around the initiative cycle to get bored or irritated. If your experienced players are also your most impatient players... well.
Eight Vs Sixteen Is A Big Battle
It's not completely unreasonable (except, see this excellent analysis about challenge ratings) but still, it is a big battle. Running big battles is also an intermediate player/GM skill just like running large parties.
This was a really heavy lift-- for you and all your players-- to be doing on what was the second session ever for you and the majority of the players. This is hard on the players (especially if the more experienced ones think the challenge rating is off) but it's really hard on you the GM. And you've seen exactly why, in your admission of making mistakes in combat.
(Don't feel too bad about that. It happens to everyone. But do recognize when you're putting yourself into situations that make that very likely.)
You can see how these things start to feed on each other, right?
Some Mild To Moderate Railroading
I can't tell from your brief description whether this is a major issue, or just the seeds of something that could later become a major issue. If your game plan starts with the assumption that an NPC is going to be kidnapped away from the PCs in a big combat, you're running at least two major risks:
The players are crit-rolling geniuses who annihilate your combatants, and with it, your plot arc.
You get so caught up in making the scene go how you want that it's no longer a game, it's a script... and trust me, sibling, your players will figure that out.
At this stage, that's just a heads-up and something to read up on in your copious free time.
Sniping And Whining
I think you bear some responsibility for this because of the first three points above-- you didn't know it, but you were putting together a crucible for your players. But I don't think you bear all of the responsibility for it. Some players are jerks. Some players are overly excitable. Some players get caught up in the moment, and even momentary setbacks hurt acutely. It's understandable, to some degree, but it does not come with an automatic apology and it's not a justification, either.
Some Candidate Solutions
Some of these can be started during a belated Session Zero; other things you have to address during play. And somethings probably can't be fixed directly, you just have to grow into the roles you're all collectively in.
Address The Tone Of The Group
This can be a hard conversation to have, but it probably needs to be done. You need to devote either your next session, or some part of it, to the questions of tone that you've brought up. You're well within your rights as a GM to put a rule into place that says "No sniping at each other for sub-optimal choices in combat or character builds."
You can do this, because you're the GM and you can stop the game when it happens and tell the snipers, "No, stop doing that." The players can't always do that for themselves without being even more disruptive.
Address The Nature Of Challenging Play
As a GM, I am firmly of the belief that a game without challenge isn't much of a game. This does not mean every fight should be a backbreaking nightmare. In fact, few of them should be that hard. But the general sequence of events should be challenging, should come with risk, should punish poor shepherding of resources, etc. Something has to be at stake or the heroes would not be necessary.
This puts the responsibility on you (and it's not easy!) to balance encounters right, and on the players to trust you a little, unless or until you flagrantly abuse that trust.
But! You and you players are mostly new and learning. It's okay to actually spell this out to them: You're not trying to kill them, you are still trying to find that sweet spot of challenge that makes things fun.
Address The Preparation Issue
This one is harder, but unless you address it, it will never change. If you have to beg your players to know their characters' abilities better, do it. One suggestion I've used are asking the players to make 3x5 cards for some of their more complex stuff. It seems demeaning, but my experience with it (player and GM) has been very positive.
But I'll warn you, some players won't or can't for whatever reason. It is really hard, in my experience, to get 100% compliance on this unless you pick your players based on that trait.
Consider Asking For Help
This one is going to be a judgment call on your part, based on your understanding of your players. But if you have two experienced players and five unexperienced ones, it's worth trying to redirect any aggression from the two experienced ones into mentoring.
When this works, it works brilliantly. What you need are experienced players who can constructively, quickly, and succinctly explain what they would do and why if asked, and then back off and let the new player make their decision. Your mentor players feel respected, your new players get some hopefully good advice and learn quicker, etc.
But I warn you, when it doesn't work, it can blow up in your face. Some people do not understand mentoring and just convert it to a different form of overbearing bullying.
And frankly, some new players won't take well to this no matter how politely it comes across. That's okay. People are different. This should be opt-in, not imposed.
This is a risk only you can judge based on your knowledge of the players.
Run Smaller Combats
At least for a while, run smaller and less challenging combats. And I mean that in all senses of the word-- smaller with fewer combatants, less challenging in terms of CR, and less challenging for you to deal with. Resist the urge to set up a grand slam and swing for the fences. Just work your way up the complexity ladder gradually.
Admit Some Fault, Ask For Feedback And Advice
If this or other answers have convinced you that you share some of the blame-- if you've been convinced to do things differently-- share this with the players. Eat a little crow. Share your self-improvement plans, even if they're modest. Ask for a little feedback. The players are invested in this enterprise, too, and a little bit of humility goes a very long way.
Do this even before you go into what you want from your players. Trust me on this.
Have A Smaller Group
Aside from your comment about one of the players soon departing, not much you can do about this. Yeah, technically you can uninvite someone just because the group is too big, but that's... harsh. Especially when you let everyone in.
Start At Level One
Not much you can do about this one, either. Unless everyone is willing to scrap the game so far (unlikely) you'll all have to grow into it.
Read An Adventure Path Or Two
Another answer suggests running an Adventure Path. I disagree mildly. It's certainly a thing people do for their first game. And it's a thing other people don't do. But what I would absolutely advise is reading through at least one that is targeted at lower levels. It will give you a much better sense of what a story arc is supposed to look like, how sequences of encounters are strung together, etc.
Don't Give Up
You had a rough session, true.
But we all have and we all will again in the future. If there's any virtue at all in making mistakes, it is learning from them.
You'll do better next time.