To make your question short, and to see if I understood it correctly, we're talking about a player who made his character a certain one and roleplays it entirely different. You added that you think that it comes from inexperience, and that he created this character after you said "no" to some "freak-character"-ideas. You want to help him roleplay the character he created.
As I see it, this problem is made from two smaller ones. The first is that he doesn't see his character as interesting because the character "is normal and normal is boring". The second is that you wanna help him understand why the way he plays the character does not fit the story-world of your game.
Helping him understand that "normal is not boring
This is the more important problem, as it stands in the basis of the entire problem. If he'll see that normal characters can be interesting his "anti-persona" will perish and he'll roleplay a normal character and not a freak one. The main trick here is to show him that normal characters are not entirely normal, i.e. "no person is like the others". In order for that to work, we need to give the character depth.
The easiest way to give depth to a character is through internal conflicts. Having goals and all is nice, but without something that blocks oneself from achieving them it is far less interesting. First thing to do is to go over his character's background and see if he implemented there an internal conflict for his character. If so, show it to him and talk with him about it. If no, sit with him and help him to come with one. The internal conflict doesn't have to be extravagant, but it needs to be there. An example one might be that he loves Vincent's sister but secretly hates Vincent himself, or another like Loves the sister but thinks that he's not good enough for there. I'll take the second one as an example for this section.
The conflict gives us a few things, a few added benefits. It gives the character 2 conflicting goals: "Get the sister and prove that I'm worthy". Now, with those two we also get a kind of an achieving-plan: "If I'll show her that I'm worthy, by getting something amazing done, she'll want me and I'll be able to get her". More than that, the character gets the knowledge that each advancement in order to achieve one goal will drive the other one to the far end.
But the first conflict is even more interesting. The character here has the knowledge that he needs the brother in order to save his lover, but he just can't stand being near the brother. He'll drive the mission onward for two reasons but he'll have doubts about his lover- if he'll marry her he'll be stuck with this brother of hers.
To make long story short, simple conflicts can show the player that even normal characters are interesting and unique. When combined with goals they force the character to take certain steps along the roads, to commit certain actions along the way, that he won't want to do but will make him doubt himself and question himself and see that his problem are far more interesting than those of every freak that he'll encounter.
Another nice way to help him see the importance of conflict is through showing him and analyzing with him certain protagonists that are normal people, from the stories and movies and series (of any form)that he likes. He'll see quite quickly that the conflicts make them interesting.
But he may say that it is not enough. For that there are a few more literary tools that might help him see why normal people are interesting. The first one is having flaws (internal or external) and the second one is using "The Ghost".
Flawed characters are characters that just like normal people aren't perfect. Those flaws can be internal (self-doubts, for example, or a mild paranoia) or they can be external (they're look frightens ordinary people, for once, or a missing hand for the other). The idea is that the character has to deal with the flaw, and one day to find the strength to overcome it. The fight for the overcoming act makes the character far more interesting. A nice example of that can be seen in The Rain Man, where he learns at the end that he can count on strangers/"dumb" persons like he's brother. Another nice example can be seen in the story of The Ugly Duckling who although looking terrible learned to acknowledge himself and to accept the way he looks, to accept his difference.
"The Ghost" is an event from the past that just like a ghost haunts the character to this day. Again, trying to cope with it is what builds a deep character. One example for this can be seen in the movie Inception, where we literally have a ghost- Cob's wife. Another example for this can be seen in the movie Casablanca, where he has to deal with his broken relationship with Ilsa. This Ghost is far more interesting as the originator of the Ghost actually comes back to his life. In Frozen we see another kind of a Ghost- the act that one feels guilty about. Elsa actually killed her sister.
All of these techniques are there for one reason- to make regular people interesting, to give depth to the characters, to make them human beings with goals and drives and psychology.
Helping him see that his character doesn't fit the world
After he understands that he doesn't have to be a freak in order to be interesting, he will be far more understandable about playing a character that fits the world. Then, try to explain to him as calmly as you can what it is in the way he played his character that doesn’t fit the world.
Explain to him that the characters are in a world where being a freak is bad, where achieving one's goals is the ideal. Each and every one for himself, as the saying goes. Give him examples from the way he played his character and analyze with him, in a one-on-one conversation where his way of acting came from. Use the background he created to illustrate to him where your problem comes from.
Then ask him what problems he has with his character, and together try to find a solution. Maybe let him be just a little bit freakish. Maybe he needs to just create a different character. This is basically between you and him. After that show the updated character to the group and get their approval.
When combining those two, you'll get a player who his far more willing to both play the character while also seeing the problems with the way he played his character before.
Combining the two solutions
When combining the two solutions you get a better player, who understands for the future also how to create regular characters that are not freaks yet far more interesting than those freaks will ever be able to be. Furthermore, you get a player who is willing to play his character as written while still making the character fit into the world. Hope any of these helped you.
Best Answer
As you say, this house rule works well for your group. A new player objecting to a house rule they don't understand is no reason for you to change it.
And they don't understand the point of the house rule. They've observed their fellow players and seen the rule's results, and are trying to adjust their actions to fit what the rule is meant to encourage — and they've misunderstood. Instead of engaging in new, creative roleplaying, they're just copying what they see other players do in a kind of cargo cult roleplaying.
This is also the source of their frustration: they think they've solved the puzzle, and they perceive you denying them the reward for solving it. They haven't understood it though, and rewarding them for missing the point will not help them integrate into your group.
Having put in effort and missed the "target" of understanding the point of the bonus experience, though, you should meet them halfway:
You need to emphasise that you're willing to help the new player get up to speed with the group. Emphasise that, because that's what the conversation should be about — not about whether the rule is useful or not (you already know it is for your group), and not whether the player gets to challenge the award (that's not their job in the group dynamic). Just take it as given that this is how it works, that's not up for discussion and don't bring it up for discussion. Move past those givens without even mentioning them, and instead move right to discussing with the player the point of it and how they can reach that point.
P.S. — Contrary to recent online RPG community fashions that resist XP rewards, this is a common and widespread way for RPGs and groups to work, and entirely functional when done fairly and impartially. Current fashion is simply that — passing fashion. Fashion is not legit grounds to tell you it's a bad way to run your game.