[RPG] How to make the extroverted brother understand that we want to roleplay in private

To make your question short, and to see if I understood it correctly, we're talking about a player who made his character a certain one and roleplays it entirely different. You added that you think that it comes from inexperience, and that he created this character after you said "no" to some "freak-character"-ideas. You want to help him roleplay the character he created.

As I see it, this problem is made from two smaller ones. The first is that he doesn't see his character as interesting because the character "is normal and normal is boring". The second is that you wanna help him understand why the way he plays the character does not fit the story-world of your game.

Helping him understand that "normal is not boring

This is the more important problem, as it stands in the basis of the entire problem. If he'll see that normal characters can be interesting his "anti-persona" will perish and he'll roleplay a normal character and not a freak one. The main trick here is to show him that normal characters are not entirely normal, i.e. "no person is like the others". In order for that to work, we need to give the character depth.

The easiest way to give depth to a character is through internal conflicts. Having goals and all is nice, but without something that blocks oneself from achieving them it is far less interesting. First thing to do is to go over his character's background and see if he implemented there an internal conflict for his character. If so, show it to him and talk with him about it. If no, sit with him and help him to come with one. The internal conflict doesn't have to be extravagant, but it needs to be there. An example one might be that he loves Vincent's sister but secretly hates Vincent himself, or another like Loves the sister but thinks that he's not good enough for there. I'll take the second one as an example for this section.

The conflict gives us a few things, a few added benefits. It gives the character 2 conflicting goals: "Get the sister and prove that I'm worthy". Now, with those two we also get a kind of an achieving-plan: "If I'll show her that I'm worthy, by getting something amazing done, she'll want me and I'll be able to get her". More than that, the character gets the knowledge that each advancement in order to achieve one goal will drive the other one to the far end.

But the first conflict is even more interesting. The character here has the knowledge that he needs the brother in order to save his lover, but he just can't stand being near the brother. He'll drive the mission onward for two reasons but he'll have doubts about his lover- if he'll marry her he'll be stuck with this brother of hers.

To make long story short, simple conflicts can show the player that even normal characters are interesting and unique. When combined with goals they force the character to take certain steps along the roads, to commit certain actions along the way, that he won't want to do but will make him doubt himself and question himself and see that his problem are far more interesting than those of every freak that he'll encounter.

Another nice way to help him see the importance of conflict is through showing him and analyzing with him certain protagonists that are normal people, from the stories and movies and series (of any form)that he likes. He'll see quite quickly that the conflicts make them interesting.

But he may say that it is not enough. For that there are a few more literary tools that might help him see why normal people are interesting. The first one is having flaws (internal or external) and the second one is using "The Ghost".

Flawed characters are characters that just like normal people aren't perfect. Those flaws can be internal (self-doubts, for example, or a mild paranoia) or they can be external (they're look frightens ordinary people, for once, or a missing hand for the other). The idea is that the character has to deal with the flaw, and one day to find the strength to overcome it. The fight for the overcoming act makes the character far more interesting. A nice example of that can be seen in The Rain Man, where he learns at the end that he can count on strangers/"dumb" persons like he's brother. Another nice example can be seen in the story of The Ugly Duckling who although looking terrible learned to acknowledge himself and to accept the way he looks, to accept his difference.

"The Ghost" is an event from the past that just like a ghost haunts the character to this day. Again, trying to cope with it is what builds a deep character. One example for this can be seen in the movie Inception, where we literally have a ghost- Cob's wife. Another example for this can be seen in the movie Casablanca, where he has to deal with his broken relationship with Ilsa. This Ghost is far more interesting as the originator of the Ghost actually comes back to his life. In Frozen we see another kind of a Ghost- the act that one feels guilty about. Elsa actually killed her sister.

All of these techniques are there for one reason- to make regular people interesting, to give depth to the characters, to make them human beings with goals and drives and psychology.

Helping him see that his character doesn't fit the world

After he understands that he doesn't have to be a freak in order to be interesting, he will be far more understandable about playing a character that fits the world. Then, try to explain to him as calmly as you can what it is in the way he played his character that doesn’t fit the world.

Explain to him that the characters are in a world where being a freak is bad, where achieving one's goals is the ideal. Each and every one for himself, as the saying goes. Give him examples from the way he played his character and analyze with him, in a one-on-one conversation where his way of acting came from. Use the background he created to illustrate to him where your problem comes from.

Then ask him what problems he has with his character, and together try to find a solution. Maybe let him be just a little bit freakish. Maybe he needs to just create a different character. This is basically between you and him. After that show the updated character to the group and get their approval.

When combining those two, you'll get a player who his far more willing to both play the character while also seeing the problems with the way he played his character before.

Combining the two solutions

When combining the two solutions you get a better player, who understands for the future also how to create regular characters that are not freaks yet far more interesting than those freaks will ever be able to be. Furthermore, you get a player who is willing to play his character as written while still making the character fit into the world. Hope any of these helped you.

Different players get different things out of gaming. Unfortunately, some people's gaming styles mean that their fun comes at the expense of others'.

Often in a case like this, the player either wants more attention than the other players or, via their outlier character, wants their character to constrain/implicitly control the rest of the group. This is unhealthy and distracting to the rest of the group, and they are justified in not liking it.

First, forward the group Making the Tough Decisions (see the "Deciding to React Differently" section) and ask them to read it. "My character would do that" is not an excuse for being a jerk. They need to find a type of character that provides their kind of fun but not at the expense of the experience of the other gamers. I had a player once who wanted to be "evil" and work against the party, so I had him be a spy and send reports back on their activity all the time, but was strictly instructed to "never blow his cover." So to the rest of the group he was perfectly helpful. The player eventually left and the PCs even still kept up with the NPC, they never found out he was a spy. This got that player his somewhat-dysfunctional fun without adverse effect on the group.

Second, the other players don't have to accept the control of the outlier character. Make it clear "we don't want that." If they do it anyway, then just have your characters react as they would in game to a weirdo freak. We had one serious game, with espionage and such, and a player brought in a new gnome character who was clinically insane. He'd just say stuff like "Turnips wheee!" and not make any sense. When he found out our secret-agent stuff, we sat him down and said "So, can you keep our secret?" "Turnips wheee!" "OK, so listen to me very carefully. If you don't start making some Goddamned sense right now we're going to tie you in a sack and drown you in the river." Our characters were not willing to have their lives in the hands of some incoherent freak.

This may lead to group exclusion. Which is sad but sometimes you have to do it. We had one player that was always "like that," and when he brought in a new character to our sci-fi game it was the last straw. Our spaceship touched down, we went recruiting crew. This guy couldn't even express why we'd hire him. "I have... skills. I can do things... that need doing." We left him on the planet. The group met later and decided to disinvite him from the group, as he just did that all the time.

@JonathanHobbs cites the Five Geek Social Fallacies, and he's right to do so. Heck, there's two of them, let them go play their own freak game together. Because groups have a right to set their playstyle. It's like recreational sports teams. Some people want to be goons and dress up in mascot costumes and play silly soccer. Other teams want to be serious and compete. Both get something different out of the hobby. But the serious team is welcome and expected to bounce a "mascot boy" from their team.

And third, we come to splitting the party. I was running games for a fairly large group, and I and some of the players wanted to do a more seriously immersive game. Half the group was on board and half wasn't. So I started an immersive game and invited those people to it, and I started a "fun" game and invited the others (eventually rotating out of GMing for the "fun" one). We had a five year long super deep immersive campaign that everyone looks back on, ten years later, as our single best roleplaying experience ever. You deserve that, and you don't have to always give in to someone that doesn't want that.