[RPG] How to handle a troublesome player even before the game starts

To make your question short, and to see if I understood it correctly, we're talking about a player who made his character a certain one and roleplays it entirely different. You added that you think that it comes from inexperience, and that he created this character after you said "no" to some "freak-character"-ideas. You want to help him roleplay the character he created.

As I see it, this problem is made from two smaller ones. The first is that he doesn't see his character as interesting because the character "is normal and normal is boring". The second is that you wanna help him understand why the way he plays the character does not fit the story-world of your game.

Helping him understand that "normal is not boring

This is the more important problem, as it stands in the basis of the entire problem. If he'll see that normal characters can be interesting his "anti-persona" will perish and he'll roleplay a normal character and not a freak one. The main trick here is to show him that normal characters are not entirely normal, i.e. "no person is like the others". In order for that to work, we need to give the character depth.

The easiest way to give depth to a character is through internal conflicts. Having goals and all is nice, but without something that blocks oneself from achieving them it is far less interesting. First thing to do is to go over his character's background and see if he implemented there an internal conflict for his character. If so, show it to him and talk with him about it. If no, sit with him and help him to come with one. The internal conflict doesn't have to be extravagant, but it needs to be there. An example one might be that he loves Vincent's sister but secretly hates Vincent himself, or another like Loves the sister but thinks that he's not good enough for there. I'll take the second one as an example for this section.

The conflict gives us a few things, a few added benefits. It gives the character 2 conflicting goals: "Get the sister and prove that I'm worthy". Now, with those two we also get a kind of an achieving-plan: "If I'll show her that I'm worthy, by getting something amazing done, she'll want me and I'll be able to get her". More than that, the character gets the knowledge that each advancement in order to achieve one goal will drive the other one to the far end.

But the first conflict is even more interesting. The character here has the knowledge that he needs the brother in order to save his lover, but he just can't stand being near the brother. He'll drive the mission onward for two reasons but he'll have doubts about his lover- if he'll marry her he'll be stuck with this brother of hers.

To make long story short, simple conflicts can show the player that even normal characters are interesting and unique. When combined with goals they force the character to take certain steps along the roads, to commit certain actions along the way, that he won't want to do but will make him doubt himself and question himself and see that his problem are far more interesting than those of every freak that he'll encounter.

Another nice way to help him see the importance of conflict is through showing him and analyzing with him certain protagonists that are normal people, from the stories and movies and series (of any form)that he likes. He'll see quite quickly that the conflicts make them interesting.

But he may say that it is not enough. For that there are a few more literary tools that might help him see why normal people are interesting. The first one is having flaws (internal or external) and the second one is using "The Ghost".

Flawed characters are characters that just like normal people aren't perfect. Those flaws can be internal (self-doubts, for example, or a mild paranoia) or they can be external (they're look frightens ordinary people, for once, or a missing hand for the other). The idea is that the character has to deal with the flaw, and one day to find the strength to overcome it. The fight for the overcoming act makes the character far more interesting. A nice example of that can be seen in The Rain Man, where he learns at the end that he can count on strangers/"dumb" persons like he's brother. Another nice example can be seen in the story of The Ugly Duckling who although looking terrible learned to acknowledge himself and to accept the way he looks, to accept his difference.

"The Ghost" is an event from the past that just like a ghost haunts the character to this day. Again, trying to cope with it is what builds a deep character. One example for this can be seen in the movie Inception, where we literally have a ghost- Cob's wife. Another example for this can be seen in the movie Casablanca, where he has to deal with his broken relationship with Ilsa. This Ghost is far more interesting as the originator of the Ghost actually comes back to his life. In Frozen we see another kind of a Ghost- the act that one feels guilty about. Elsa actually killed her sister.

All of these techniques are there for one reason- to make regular people interesting, to give depth to the characters, to make them human beings with goals and drives and psychology.

Helping him see that his character doesn't fit the world

After he understands that he doesn't have to be a freak in order to be interesting, he will be far more understandable about playing a character that fits the world. Then, try to explain to him as calmly as you can what it is in the way he played his character that doesn’t fit the world.

Explain to him that the characters are in a world where being a freak is bad, where achieving one's goals is the ideal. Each and every one for himself, as the saying goes. Give him examples from the way he played his character and analyze with him, in a one-on-one conversation where his way of acting came from. Use the background he created to illustrate to him where your problem comes from.

Then ask him what problems he has with his character, and together try to find a solution. Maybe let him be just a little bit freakish. Maybe he needs to just create a different character. This is basically between you and him. After that show the updated character to the group and get their approval.

When combining those two, you'll get a player who his far more willing to both play the character while also seeing the problems with the way he played his character before.

Combining the two solutions

When combining the two solutions you get a better player, who understands for the future also how to create regular characters that are not freaks yet far more interesting than those freaks will ever be able to be. Furthermore, you get a player who is willing to play his character as written while still making the character fit into the world. Hope any of these helped you.

Tell them what the problem is, give suggestions on how to fix it, define a set of rules, and if they don't follow kick them out.

First, set expectations. Good gaming practice is to have a session zero that goes over not just house rules, tone, theme, character building, but also table rules, like 'no cellphones' or 'keep chatter to a min'. It's too late to do that this time, but not too late to have a conversation.

Before the next game say you want to have a discussion about table behavior. Perhaps do it online since these players don't show up regularly. Bring up the problems that are happening in the game(though do your best not to make it an attack on these players, because that increases the chance they'll get defensive and it will end in an argument). Explain that you want the game to run smoothly and in order to do that, you need X, where X is a certain amount of agreed attendance (or a way to mitigate lack of attendance, such as characters that are designed to disappear and reappear and are OK not knowing whole swathes of plot) and a certain respect for the game at the table. Suggest what rules you're OK with and elicit feedback. Make sure you don't end the discussion without saying, "These are the rules and this is what I'm holding you to." Total agreement is the goal but since this appears to be your game it's not necessary. Just make sure you have a reasonable set of rules most of the group is OK with.

When people start breaking the rules, pull them aside and mention it privately later. During the game simply ask them to tighten up a bit. If it's rare, give them more chances. If it's frequent, boot them when they have broken the rules 3 times or so, depending on how severely disruptive they are. If they kill a whole game session after the talk, once is enough. You want to make sure you give them a chance, recognizing that they may be trying but not good at it. One of the problems with RPG's is that they are also often friend hangout time and most people want to relax and enjoy hanging, not do serious 'game face' acting the whole time. There needs to be a balance but that includes recognizing that slipping into bad gaming behaviors is an easy habit to acquire. The important part is that they are trying, getting better, and not being overly disruptive. They need to show respect for the game and those who want to take it seriously as well.

This is the rule I follow whenever I run a game; it works very well. My group is particularly bad at showing up and not being disruptive. When other people step into the GM role it tends to be much worse, sometimes we lose whole sessions doing almost nothing. Because I set expectations no one is mad when I enforce them. Because I talk to people separately they are less likely to be defensive and more receptive to my point of view. I've never had to kick anyone out either. They will generally leave on their own if they can't follow the rules because they aren't having the type of fun they want; it essentially becomes a mutual break up. That may not work in your case depending on how mature these players are but mine are hardly the most emotionally developed.