My suggestion:
Mirror PC stats, abilities and numbers to give them a challenge. If you have a fighter, wizard and rogue group, and you want to put them up against monsters, you'll need to set them against something tanky, something ranged, and something that can hit hard and fast.
Alternately, you could design special monsters like a Colossal Scorpion. Give the torso the tanky stats, and make each pincer a rogue type damage dealer. In addition, the stinger could be considered limited ranged (30'/60'). This would give the scorpion 3 distinct parts that can be targetted, as well as a main body that will kill the whole thing.
You can apply this to any set of monster enemies as well. An Ogre, axe throwing orc, and orc shaman would be a good trio to set up against PC's if you balance their stats according to the PC stats.
The problem you're going to be facing with trying to balance this is something you've already pointed out, the CR ratings don't mesh with single fights. It will take a good amount of experimentation to get a feel for it, but if you start with PC stats and incorporate those into monsters, you should be fine using that as a baseline.
Numbers version of level 6 characters:
Fighter - 65 HP - 2 handed weapon
Wizard - 32 HP - evocation
Rogue - 40 HP - Whip, throwing knives
Creature battle:
Colossal Scorpion +7 to hit (Mutli-attack, this creature attacks with two claw attacks and a stinger attack)
Legendary Resistance 1 - Once per encounter, this creature can turn a failed save into a success
Regenerate - As a bonus action, the scorpion can sacrifice 40 Torso HP to fully regenerate a destroyed claw or stinger. That claw or stinger only becomes available for use on the creatures next turn.
Torso - 150 HP - damage resistant non-magical
Left Claw - 40 HP - Pincer attack 1d10 + 3
Right Claw - 40 HP - Crushing Grip 1d10 +3, auto-grapples
Stinger - 40 HP - Acidic spray, ranged 30'/60' 1d8 + 4 acid damage
If you kill the claws and stinger, the scorpion dies. If you kill the torso, the scorpion dies.
That's just one suggested battle to keep things interesting. The main point is that a battle like that enables a monster to take as many actions as your players can, and it doesn't destroy the creatures ability to remain a challenge.
If you break the enemy down into trios as well, consider giving them similar abilities to player characters. This will ensure the challenge rating is appropriate. Try to avoid making the mistake of jacking up their AC, to hit bonus, HP and resistances. Keep it consistent with what the PC's have to maintain the arena feel to the tournament.
You've already done what you can for that particular situation
The rest is up to those two.
I tried to tell him not take it seriously and that he acted was wrong
but he tuned me out. I know that the fighter feels terrible because
he, and everyone playing, didn't realize how important the last kill
was to him. So before I left I told him that I understand that he is
angry but he should call the fighter. I explained that maybe if he
express his feelings then the two of them could comprise or come to an
understanding.
If those two cannot, after time to settle down, make peace with each other then their time together in the party is going to be short. D&D 5e is generally a team game. It is a game of taking turns (basic stuff from kindergarden) and sharing in team success.
I plan to call both the fighter and other player to see how they are feeling and their concerns on the matter.
Since most of the advice on this site about player conflict is "talk to the players" you are already on the right path. You may be wondering about what the next few sessions will hold for you, since you want this to be fun and not interpersonal drama.
What can I as a DM do to prevent problems like this?
Make sure everyone gets their turn. (In a mechanical sense, stick to the initiative order, and make sure that the players know when it is their turn).
- @umbranus made this point in a comment:
... I suggest using initiative to solve stuff like that. The Fighter used his turn to throw the barrel, so everyone else should get a turn before the Fighter goes again1.
Make sure each player gets some time in the spotlight.
Remind everyone that we play these games for fun.
The link to the kill steal problem that @DaleM provided is worth reviewing before your next session.
1 If the reason that the Rogue was unable to take his action is that you gave the Fighter a turn (tossing the barrel) and then he got to take his chance before the Rogue did, then there's some DM fault in this for not keeping the initiative/turn order going. However, if they were all over the room and the Fighter was closer, or was able to get there first in the regular turn order, then you have a different problem to deal with, as above.
Best Answer
Get a second GM
I have been in a game where there was a three-way fight. One team was the players' party. Our GM ran the second team. And our GM brought in her partner (also an experienced GM) to run the third team.
There are a couple of benefits to having another person run the third team. One major benefit is that it reduces the load on you as the GM since you only have to control half as many NPCs. This helps streamline combat somewhat.
Another very important benefit is that this third team will be able to act truly independently. If you run both teams, you will be subconsciously (or consciously) colluding with yourself and the three-way battle won't feel as authentic. The different play-style of your assistant GM will be evident to your players and this extra set of NPCs will feel different to if you had run them. And by having another person running the third team, there will be the added thrill of all three sides genuinely fighting to win.
Have rich goals
This fight will be long and slow, even with a second GM. As such, you need to make the fight narratively rewarding and more engaging than 'I hit them until they die'. Enkryptor was spot-on when he said that every combat should have a goal.
In my particular example, the goal of all three parties was to 'get the girl', a plot-critical NPC who was a long-time member of our party (so we were protecting her while the other parties both tried to take her for themselves). Because we had a goal which was more complicated than simply 'kill everything' the combat was more dynamic and exciting. There was the added complication that no side wanted the girl to die, even though the girl was quite capable of fighting back, so extra care had to be taken. It was a goal that we the players were quite invested in, which motivated us through the battle. And it meant that some NPCs fled when heavily wounded and left their underlings take care of the mission, which made the battle more authentic than people pointlessly fighting to the death.
The objectives of the three factions should also be such that there can be only one winner. If, as in your example, the goal of both the party and the secondary antagonist is to kill the main antagonist, then the most logical arrangement is for the secondary antagonist to temporarily side with the players, reducing the battle to a two-sided one. However, in my scenario, only one team could get the girl, so we all fought tooth and nail to make sure it was ours. There could be no parley, not without losing her.
Allow lots of time
This battle will take an entire session. It will likely be a long session. Plan accordingly. You should end the previous session just as you've set up the battle and combat seems inevitable, even if that means ending early, otherwise you will not have time for your planned battle. This also builds up the hype for your players.
Abstract away unimportant creatures
In such a large battle, you are likely to have mooks fighting mooks. However, your players don't care about the mooks, they care about themselves and the major NPCs.
You can substantially reduce the number of active characters in the encounter while maintaining the scale of the battle by going "this group of soldiers is fighting this other group of soldiers and they are both busy doing that" with an implicit "they won't bother you if you don't bother them". The outcome can either be determined as is narratively convenient or with a few quick dice rolls, although the result should be withheld until either the end of the battle or when it becomes relevant. This puts greater emphasis on the characters which matter while keeping a cinematic feel to your epic battle.
(This advice applies to all battles with many NPCs, not just three-sided ones.)
Alternatives
I recommend getting a second GM to run the extra faction. I recommend that each team have mutually exclusive goals which means that they all must fight against each other, and that these goals should be more elaborate than "I want everyone else to die".
If none of these options work for you, you may need to reconsider doing a true three-way battle.
You might want to run the NPC vs NPC battle beforehand and let the players intervene when they see fit (if it is appropriate for the players to ambush the NPCs).
You might have the players fight just the main antagonist, then when the battle is almost over the secondary antagonist swoops in and ambushes both parties. This makes it a two phase battle. Because one NPC team will be mostly (or entirely) dead by this point, the second phase would run quite similarly to a normal two-sided battle.
Or, if the narrative allows, you might have the secondary antagonist make a temporary truce with the players. Then the battle will be reduced to two factions and end up being simpler to run.