I'm a relatively new DM. I put together a group of 5 all new players who all showed interest and we have a pretty good group. One of the players decided to play a cocky rogue type, which is absolutely fine since he always plays it in character and doesn't go OOC much at all. One of our other players was playing a stern serious paladin but recently he's been bantering with the cocky rogue type. Unfortunately for this he breaks character and does a lot of meta-referencing.
Despite my best efforts to try and calm them down so I can describe scenes or another player can make their turn they continue to force banter which is having a negative effect on my enjoyment of the game and the other three players. In our previous session I was essentially describing the scenes to the three players who take it seriously and the other two just tuned in afterwards and asked questions about the situation I had just described.
How can I politely nudge them into paying more attention and sticking to their characters?
Best Answer
Pick Your Battles
Let some things slide. Decide if it's worth it to call him out each time. Understand that jokes are a part of every game. It's part of the social event of roleplaying. It means your players are having fun, and disruptions aside, that is a good thing. Let it slide if the action at the table is more light-hearted (e.g. having a drink in a tavern). Enforce serious roleplaying when serious things are happening (e.g. telling the evil overlord his reign of terror has come to an end).
Tell Him What You Told Us
Discuss it with your player. Explain why it's disruptive. Get him to agree to tone down the jokes. Do so one-on-one (praise in public, "punish" in private) so as not to embarrass him.
Jokes Are Sometimes A Sign of Discomfort
I've found that the players who joke the most are the most uncomfortable roleplaying. Jokes are a way for them to cope with how uncomfortable they feel expressing themselves as a character. This is particularly true in serious games, especially horror games. In the case of horror games, it may not be so much discomfort with expressing themselves, as it is discomfort with the subject matter. Both are worth addressing in your conversation with your player. Is he new to roleplaying? Is the subject matter of your game making him uncomfortable?
Have NPCs take him seriously
Essentially, to enforce roleplaying, treat every statement as if it was said in character. Have the NPCs respond to his meta-references as if he had said something really confusing. Don't harp on it, like having NPCs hold a grudge for something stupid he said, just have them always engage in character. Listen to the Adventure Zone. Although it's a light-hearted campaign, Griffin does a really good job of reigning the silliness in. The same techniques can be used in a more serious campaign.
Give Bennies
Find out what your rpg system uses as a reward, and hand those out as carrots to good roleplayers. In D&D 5e this is Inspiration. In Fate, this is Fate points. Only use things as carrots that you can safely use without affecting game balance. In D&D 5e, for example, there are ways to award XP for good roleplaying, but if you overdo it, you might level them up too fast. Inspiration, on the other hand, is fairly innocuous and only gives temporary advantage on a roll.
When players see what other people are getting rewarded for, they will behave in a way that gets them the reward. It's a good idea to hand out rewards as soon as they demonstrate the desired behavior so as to reinforce what they did right (rather than waiting til the end of the session). Of course, as good roleplaying becomes the norm, it should become more and more difficult to get those bennies.