OK, I don't have time to answer this as I want to. My background is in psychology, and I fell into role playing games when I turned 10 in 1976. So by the time I was in college, understanding where the term Roleplaying game really came from, I understood the critical nature of immersion, how it is the most important ingredient for game success.
And to be clear, the definition of immersion is to "Immerse oneself into the identity and Role of the part one is playing. To respond, as much as possible, as the person one is playing, not as oneself."
And before getting into the smaller details, I will dive right into the fact that the very system/game one chooses has a huge amount to do with the amount of Immersion.
Metagaming is the opposite of immersion. You use both terms, but I need to make that absolute definition from the beginning. This also means rules that encourage metagaming decrease the immersion in a game and therefore, decrease the main ingredient of a roleplaying game. The mechanics are called "Dissociated Mechanics", a term coined by Justin Alexander. This is very worth reading, because it gets into many of the larger picture issues with players being able to use in-game logic to see the world around them, as opposed to the rules forcing dissociation from in-game logic.
Once the players assume that rules are going to determine the content of an encounter or treasure (based on EL, or whatever) instead of what the environment or history of the area dictate, verisimilitude is lost.
Vreeg's Rules of Setting design are also heavily immersion related. My current campaign is 26 or so years old (started in '83). Building verisimilitude is a huge part of this.
Vreeg's first Rule of Setting Design
Make sure the ruleset you are using
matches the setting and game you want
to play, because the setting and game
WILL eventually match the system.
Corollary to Vreeg's First Rule
The proportion of rules given to a
certain dimension of an RPG partially
dictate what kind of game the rules
will create. If 80% of the rulebook
is written about thieves and the
underworld, the game that is meant
for is thieving. If 80% of the
mechanics are based on combat, the
game will revolve around combat.
- Multiply this by 10 if the reward
system is based in the same area as
the preponderance of rules.
2nd Corollary
Character growth is
the greatest reinforcer. The
synthesis of pride in achievement
with improvement in the character
provides over 50% of the
reinforcement in playing the game.
Rules that involve these factors are
the most powerful in the game.
Vreeg’s Second Rule of Setting Design
Consistency is the
Handmaiden of Immersion and
Verisimilitude. Keep good notes, and
spend a little time after every
creation to ‘connect the dots’. If
you create a foodstuff or drink, make
sure you note whether the bars or inns
the players frequent stock it. Is it made
locally, or is it imported? If so,
where from? If locally made, is it
exported?
Vreeg's Third Rule of Setting Design
The World In Motion is critical
for Immersion, so create 'event
chains' that happen at all levels of
design. The players need to feel like
things will happen with or without
them; they need to feel like they can
affect the outcome, but event-chains
need velocity, not just speed.
Vreeg's Fourth Rule of Setting Design
Create motivated events and
NPCs, this will invariably create
motivated PCs. Things are not just
happening, they happen because they
matter to people (NPCs). There is no
need to overact, just make sure that the
settings and event-chains are
motivated and that the PCs feel
this.
Vreeg's Fifth Rule of Setting Design
The Illusion of Preparedness is critical
for immersion; allowing the players to see
where things are improvised or changed
reminds them to think outside the setting,
removing them forcibly from immersion.
Whenever the players can see the hand of the GM - even when the GM needs to change things in their favor -
it removes them from the immersed position.
(Cole, of the RPGsite, gets credit for the term).
Remember that part of immersion is the lack of feeling walls around and rails under the characters. This means that the players should not feel that there are things that their character cannot do solely because of the rules or the GM's mindset. The job of the GM is to enable roleplay, not to inhibit it.
This also means the GM must be as immersed as the players, or more.
Another big-picture thing that may irk some folk who sell stuff is that published settings can hurt immersion. They don't destroy it; but when the players have a lot of knowledge about a setting that their character would not have, this increases the opportunity to use it, consciously or unconsciously. Similarly, if your setting has its own bestiary that the characters learn as they go along, or at least a lot of homebrew tweaks, the players get used to working with the in-house data and not trusting the published sources.
If you have done all of this larger-scope stuff, the smaller scope stuff becomes easier. As a GM with miles on the tires, I find that playing up the level of knowledge my NPCs might have and do not have helps keep the players in the same mindset. Players key heavily off the way the GM plays their NPCs. They won't do the funny voices or the mannerisms if the GM does not, and if the GM is particularly careful about what their NPCs know and don't know, especially verbally, the players emulate this.
To make your question short, and to see if I understood it correctly, we're talking about a player who made his character a certain one and roleplays it entirely different. You added that you think that it comes from inexperience, and that he created this character after you said "no" to some "freak-character"-ideas. You want to help him roleplay the character he created.
As I see it, this problem is made from two smaller ones. The first is that he doesn't see his character as interesting because the character "is normal and normal is boring". The second is that you wanna help him understand why the way he plays the character does not fit the story-world of your game.
Helping him understand that "normal is not boring
This is the more important problem, as it stands in the basis of the entire problem. If he'll see that normal characters can be interesting his "anti-persona" will perish and he'll roleplay a normal character and not a freak one. The main trick here is to show him that normal characters are not entirely normal, i.e. "no person is like the others". In order for that to work, we need to give the character depth.
The easiest way to give depth to a character is through internal conflicts. Having goals and all is nice, but without something that blocks oneself from achieving them it is far less interesting. First thing to do is to go over his character's background and see if he implemented there an internal conflict for his character. If so, show it to him and talk with him about it. If no, sit with him and help him to come with one. The internal conflict doesn't have to be extravagant, but it needs to be there. An example one might be that he loves Vincent's sister but secretly hates Vincent himself, or another like Loves the sister but thinks that he's not good enough for there. I'll take the second one as an example for this section.
The conflict gives us a few things, a few added benefits. It gives the character 2 conflicting goals: "Get the sister and prove that I'm worthy". Now, with those two we also get a kind of an achieving-plan: "If I'll show her that I'm worthy, by getting something amazing done, she'll want me and I'll be able to get her". More than that, the character gets the knowledge that each advancement in order to achieve one goal will drive the other one to the far end.
But the first conflict is even more interesting. The character here has the knowledge that he needs the brother in order to save his lover, but he just can't stand being near the brother. He'll drive the mission onward for two reasons but he'll have doubts about his lover- if he'll marry her he'll be stuck with this brother of hers.
To make long story short, simple conflicts can show the player that even normal characters are interesting and unique. When combined with goals they force the character to take certain steps along the roads, to commit certain actions along the way, that he won't want to do but will make him doubt himself and question himself and see that his problem are far more interesting than those of every freak that he'll encounter.
Another nice way to help him see the importance of conflict is through showing him and analyzing with him certain protagonists that are normal people, from the stories and movies and series (of any form)that he likes. He'll see quite quickly that the conflicts make them interesting.
But he may say that it is not enough. For that there are a few more literary tools that might help him see why normal people are interesting. The first one is having flaws (internal or external) and the second one is using "The Ghost".
Flawed characters are characters that just like normal people aren't perfect. Those flaws can be internal (self-doubts, for example, or a mild paranoia) or they can be external (they're look frightens ordinary people, for once, or a missing hand for the other). The idea is that the character has to deal with the flaw, and one day to find the strength to overcome it. The fight for the overcoming act makes the character far more interesting. A nice example of that can be seen in The Rain Man, where he learns at the end that he can count on strangers/"dumb" persons like he's brother. Another nice example can be seen in the story of The Ugly Duckling who although looking terrible learned to acknowledge himself and to accept the way he looks, to accept his difference.
"The Ghost" is an event from the past that just like a ghost haunts the character to this day. Again, trying to cope with it is what builds a deep character. One example for this can be seen in the movie Inception, where we literally have a ghost- Cob's wife. Another example for this can be seen in the movie Casablanca, where he has to deal with his broken relationship with Ilsa. This Ghost is far more interesting as the originator of the Ghost actually comes back to his life. In Frozen we see another kind of a Ghost- the act that one feels guilty about. Elsa actually killed her sister.
All of these techniques are there for one reason- to make regular people interesting, to give depth to the characters, to make them human beings with goals and drives and psychology.
Helping him see that his character doesn't fit the world
After he understands that he doesn't have to be a freak in order to be interesting, he will be far more understandable about playing a character that fits the world. Then, try to explain to him as calmly as you can what it is in the way he played his character that doesn’t fit the world.
Explain to him that the characters are in a world where being a freak is bad, where achieving one's goals is the ideal. Each and every one for himself, as the saying goes. Give him examples from the way he played his character and analyze with him, in a one-on-one conversation where his way of acting came from. Use the background he created to illustrate to him where your problem comes from.
Then ask him what problems he has with his character, and together try to find a solution. Maybe let him be just a little bit freakish. Maybe he needs to just create a different character. This is basically between you and him. After that show the updated character to the group and get their approval.
When combining those two, you'll get a player who his far more willing to both play the character while also seeing the problems with the way he played his character before.
Combining the two solutions
When combining the two solutions you get a better player, who understands for the future also how to create regular characters that are not freaks yet far more interesting than those freaks will ever be able to be. Furthermore, you get a player who is willing to play his character as written while still making the character fit into the world. Hope any of these helped you.
Best Answer
I do.
I'm so competitive I managed to win a game of Fiasco (a very non-competitive game).
Luckily, I know why you feel this way and where the source of the problem is.
Unfortunately, D&D 3.X is more often than not the cause of this dicothomy.
There's a thing game designers call reward cycle: encouraging the players to behave in a certain manner by giving them some mechanical advantage if/when they do.
This is done differently in different games where the authors are conscious about the need to encourage the intended behavior with positive reinforcement. Some examples:
These games make bad choices for your character become good choices for you, encouraging you to play your character's flaws as well as his qualities.
D&D 3.5 also has a reward cycle, but it's not something the authors appear to have planned, unless you want to believe D&D authors actually wanted to encourage playing some psychopats who only care about money and xp.
Emphasis is because only killing/defeating/avoiding monsters has defined and quantified rules governing the gains. More than that, most of the things you can buy with money or XP are useful to be a better monsterslayer.
In previous editions, you gained XP based on the treasure you could get your hands on: avoiding confrontations meant suffering less HP (or character) losses. Now, the easy way to solve an encounter is "just kill everything". This means in a D&D game there's a straight path to "winning" the game that requires you to bash monsters in the most efficient way. The most efficient way, as many movie villains will tell you, is to get rid of all emotions and vulnerabilities. Weakness leads to being killed. Dying makes you lose some of the resources you gathered.
Of course, this does not produce a fiction that's satisfying to those who want to tell a "realistic" story, nor to those who want high fantasy, epic or similar results.
What usually happens in a D&D game where being a party of murderhoboes is not the intended result is that people is expected to behave consistently (and rolpelay an actual human being), and whoever can't find where the line lies and balance on it is usually either despised for "not being able to roleplay" or is willingly taking risks and worsening his chances at suceeding.
Someone calls it "role-playing vs. roll-playing".
It might be arguable that an equilibrium point or where none of the two happens exists.
So, what do you do?
@Miniman wrote a comment where he suggested to roleplay a character who wants to make optimal choices himself. I suggest you don't: more often than not, being stuck on finding the optimal choice in-character is a flaw you'd probably try to instinctively avoid, especially when metaplaying is involved and you know a stupid decision of your character would make for a great strategy.
What you don't want to get is a character that behaves in a completely different way in different situations, because the lack of internal coherency hampers the kind of immersion you're currently looking for.