To make your question short, and to see if I understood it correctly, we're talking about a player who made his character a certain one and roleplays it entirely different. You added that you think that it comes from inexperience, and that he created this character after you said "no" to some "freak-character"-ideas. You want to help him roleplay the character he created.
As I see it, this problem is made from two smaller ones. The first is that he doesn't see his character as interesting because the character "is normal and normal is boring". The second is that you wanna help him understand why the way he plays the character does not fit the story-world of your game.
Helping him understand that "normal is not boring
This is the more important problem, as it stands in the basis of the entire problem. If he'll see that normal characters can be interesting his "anti-persona" will perish and he'll roleplay a normal character and not a freak one. The main trick here is to show him that normal characters are not entirely normal, i.e. "no person is like the others". In order for that to work, we need to give the character depth.
The easiest way to give depth to a character is through internal conflicts. Having goals and all is nice, but without something that blocks oneself from achieving them it is far less interesting. First thing to do is to go over his character's background and see if he implemented there an internal conflict for his character. If so, show it to him and talk with him about it. If no, sit with him and help him to come with one. The internal conflict doesn't have to be extravagant, but it needs to be there. An example one might be that he loves Vincent's sister but secretly hates Vincent himself, or another like Loves the sister but thinks that he's not good enough for there. I'll take the second one as an example for this section.
The conflict gives us a few things, a few added benefits. It gives the character 2 conflicting goals: "Get the sister and prove that I'm worthy". Now, with those two we also get a kind of an achieving-plan: "If I'll show her that I'm worthy, by getting something amazing done, she'll want me and I'll be able to get her". More than that, the character gets the knowledge that each advancement in order to achieve one goal will drive the other one to the far end.
But the first conflict is even more interesting. The character here has the knowledge that he needs the brother in order to save his lover, but he just can't stand being near the brother. He'll drive the mission onward for two reasons but he'll have doubts about his lover- if he'll marry her he'll be stuck with this brother of hers.
To make long story short, simple conflicts can show the player that even normal characters are interesting and unique. When combined with goals they force the character to take certain steps along the roads, to commit certain actions along the way, that he won't want to do but will make him doubt himself and question himself and see that his problem are far more interesting than those of every freak that he'll encounter.
Another nice way to help him see the importance of conflict is through showing him and analyzing with him certain protagonists that are normal people, from the stories and movies and series (of any form)that he likes. He'll see quite quickly that the conflicts make them interesting.
But he may say that it is not enough. For that there are a few more literary tools that might help him see why normal people are interesting. The first one is having flaws (internal or external) and the second one is using "The Ghost".
Flawed characters are characters that just like normal people aren't perfect. Those flaws can be internal (self-doubts, for example, or a mild paranoia) or they can be external (they're look frightens ordinary people, for once, or a missing hand for the other). The idea is that the character has to deal with the flaw, and one day to find the strength to overcome it. The fight for the overcoming act makes the character far more interesting. A nice example of that can be seen in The Rain Man, where he learns at the end that he can count on strangers/"dumb" persons like he's brother. Another nice example can be seen in the story of The Ugly Duckling who although looking terrible learned to acknowledge himself and to accept the way he looks, to accept his difference.
"The Ghost" is an event from the past that just like a ghost haunts the character to this day. Again, trying to cope with it is what builds a deep character. One example for this can be seen in the movie Inception, where we literally have a ghost- Cob's wife. Another example for this can be seen in the movie Casablanca, where he has to deal with his broken relationship with Ilsa. This Ghost is far more interesting as the originator of the Ghost actually comes back to his life. In Frozen we see another kind of a Ghost- the act that one feels guilty about. Elsa actually killed her sister.
All of these techniques are there for one reason- to make regular people interesting, to give depth to the characters, to make them human beings with goals and drives and psychology.
Helping him see that his character doesn't fit the world
After he understands that he doesn't have to be a freak in order to be interesting, he will be far more understandable about playing a character that fits the world. Then, try to explain to him as calmly as you can what it is in the way he played his character that doesn’t fit the world.
Explain to him that the characters are in a world where being a freak is bad, where achieving one's goals is the ideal. Each and every one for himself, as the saying goes. Give him examples from the way he played his character and analyze with him, in a one-on-one conversation where his way of acting came from. Use the background he created to illustrate to him where your problem comes from.
Then ask him what problems he has with his character, and together try to find a solution. Maybe let him be just a little bit freakish. Maybe he needs to just create a different character. This is basically between you and him. After that show the updated character to the group and get their approval.
When combining those two, you'll get a player who his far more willing to both play the character while also seeing the problems with the way he played his character before.
Combining the two solutions
When combining the two solutions you get a better player, who understands for the future also how to create regular characters that are not freaks yet far more interesting than those freaks will ever be able to be. Furthermore, you get a player who is willing to play his character as written while still making the character fit into the world. Hope any of these helped you.
As you say, this house rule works well for your group. A new player objecting to a house rule they don't understand is no reason for you to change it.
And they don't understand the point of the house rule. They've observed their fellow players and seen the rule's results, and are trying to adjust their actions to fit what the rule is meant to encourage — and they've misunderstood. Instead of engaging in new, creative roleplaying, they're just copying what they see other players do in a kind of cargo cult roleplaying.
This is also the source of their frustration: they think they've solved the puzzle, and they perceive you denying them the reward for solving it. They haven't understood it though, and rewarding them for missing the point will not help them integrate into your group.
Having put in effort and missed the "target" of understanding the point of the bonus experience, though, you should meet them halfway:
"It's clever how you noticed what the other players did to earn the bonus experience. That's great, but it's only halfway to getting the point of the bonus. The bonus is a reward for individual creativity in roleplaying, so simple copying doesn't earn the bonus.
"Does that make sense? This is an important tool that has worked for us, and I want you to understand what it's for so that it works for you too."
You need to emphasise that you're willing to help the new player get up to speed with the group. Emphasise that, because that's what the conversation should be about — not about whether the rule is useful or not (you already know it is for your group), and not whether the player gets to challenge the award (that's not their job in the group dynamic). Just take it as given that this is how it works, that's not up for discussion and don't bring it up for discussion. Move past those givens without even mentioning them, and instead move right to discussing with the player the point of it and how they can reach that point.
P.S. — Contrary to recent online RPG community fashions that resist XP rewards, this is a common and widespread way for RPGs and groups to work, and entirely functional when done fairly and impartially. Current fashion is simply that — passing fashion. Fashion is not legit grounds to tell you it's a bad way to run your game.
Best Answer
It depends on your table, so ask your table; here are some different ways I've run it at my own
There are a few different ways I've tackled the "RP vs. skill level problem" and I usually just ask the group which they'd prefer unless a campaign demands one method over another:
RP determines everything, no skill checks exist
Note that this doesn't completely remove skill proficiencies from being helpful; they may still have in-combat uses or some sort of other feature might call for a skill check, but this is far from an ideal option. (You may end up having to rearrange class proficiencies as many end up being near-useless)
RP determines the result, skill check determines any additional circumstances
Here I have the character (player) RP and this (depending on how well they do it) determines what the result of the NPC encounter is. I then have them make a check to see whether it goes off with some extra stipulations, perhaps nothing goes wrong, or the NPC throws in an extra incentive. One benefit of this style is that it prevents players from having to RP characters who are less, or even worse more persuasive/whatever than themself. An often difficult thing to do, but one that can be done.
Skill check determines the result, RP determines additional circumstances
This is basically the above but reversed, putting less pressure on the Player's ability to perform the skill as RP and more of fate is left up to the dice and the character's own proficiency in said skill.
Skill check determines everything, RP simply follows
Here the player (though not the character) already knows whether they succeed or fail, which gives them a bit of an idea of what to be arriving for. They can model how well they argue based off of the roll, some tables have preferred this method but it certainly teeters on the edges of meta-gaming and is hard to get a good balance for. It also still promotes having a high skill level as this increases the odds of a good result, and even is the only way to get a great result.
Skill level determines possible results
This is the method I most often use though it is unusual. Here I have thresholds like the following: if your passive persuasion is at least 16, you can get the great result, otherwise you can only get a good result. This method leaves less up the rolls of the dice, and gives a greater benefit to simply having a greater proficiency in some given skill. You'll note that I've left out how to actually determine the result and that's because when using this method I always ask my table how they want this to work. Sometimes there is no skill check at all, sometimes the RP is just for flavor and the passive score dictates everything (though this has the negative side-effect of promoting just having one party member being good at each skill and thus nearly-always succeeding at any sort of skillful endeavor).
Overall you've got a lot of options, including many besides the ones here, for how to handle RP and skill checks. Sometimes, unavoidably, they won't line up or really agree with each other at all, and I suggest discussing what to do in this sort of scenario with your players. If the RP is phenomenal and the dice give you the absolute worst roll perhaps some sort of bonus is in order? One thing you could do is give advantage to a check if the RP is particularly good, I've done this before as a inspiration-esque method of promoting quality RP from my players.
There are many other things to consider as well, such as quality of RP, creativity, and other less quantifiable and more qualitative aspects of the game. The bottom line (for me) is to ask your players what they want to see in the game.