The Players May Not Want To
Part of fantasy role playing for a lot of people is being able to be larger than life for a bit. They may not want their characters to feel fear at all.
Now, in a novel this may be a bad thing, since a character that isn't believable can disrupt the suspension of disbelief. But in an RPG its not necessarily a bad thing to let the players through their characters simply feel fearless and powerful, even in the face of overwhelming odds (which in your example weren't horribly overwhelming since they won.).
Their reactions might not be quite that unbelievable after all
People can keep their fear under control. When I was in the Army, I was in an Airborne unit and we did jump training frequently. I was scared every time. I still jumped out of the plane, every time. I was lucky enough that I never got in a close quarters fire fight, but plenty of people in my unit did. Not a single one ran in the time I was with that unit. Many of them did readily admit they were scared. They didn't run and the few times I heard about screaming involved people with serious injuries.
Remember, we aren't talking about some comfortable bookkeep that has never even been in a fist fight. Even a first level adventurer has mentally prepared for battle, equiped themselves for battle, and trained for battle. By the time they have added a few levels they have seen the horrors of battle and possibly the terrors of war. They know how to stand their ground.
Sure, a good role-player, when their character faces a new, powerful threat, might add details like, "My eyes widen in terror, and I feel my heart pounding in my chest." But, especially if they have a good reason to stand their ground (like a young child they must protect...), its not actually that unrealistic for them to follow with, "But I swallow my fear, and yank my sword out of its scabard!"
Remember that 300 was dramatized and fictionalized, but it was based on a real event. 300 Spartans (backed up by 700 Thespians and 400 Thebans in real history, so about 1400) stood their ground against genuinely overwhelming odds. The Persian Army was somehwere in the neighborhood of 300,000. With odds of nearly 300 to one, the Spartans knew that they were waiting for death (indeed very nearly all of them were killed). I suspect close to all of them were terrified, most of them were young men with little experience. They stood their ground.
To make your question short, and to see if I understood it correctly, we're talking about a player who made his character a certain one and roleplays it entirely different. You added that you think that it comes from inexperience, and that he created this character after you said "no" to some "freak-character"-ideas. You want to help him roleplay the character he created.
As I see it, this problem is made from two smaller ones. The first is that he doesn't see his character as interesting because the character "is normal and normal is boring". The second is that you wanna help him understand why the way he plays the character does not fit the story-world of your game.
Helping him understand that "normal is not boring
This is the more important problem, as it stands in the basis of the entire problem. If he'll see that normal characters can be interesting his "anti-persona" will perish and he'll roleplay a normal character and not a freak one. The main trick here is to show him that normal characters are not entirely normal, i.e. "no person is like the others". In order for that to work, we need to give the character depth.
The easiest way to give depth to a character is through internal conflicts. Having goals and all is nice, but without something that blocks oneself from achieving them it is far less interesting. First thing to do is to go over his character's background and see if he implemented there an internal conflict for his character. If so, show it to him and talk with him about it. If no, sit with him and help him to come with one. The internal conflict doesn't have to be extravagant, but it needs to be there. An example one might be that he loves Vincent's sister but secretly hates Vincent himself, or another like Loves the sister but thinks that he's not good enough for there. I'll take the second one as an example for this section.
The conflict gives us a few things, a few added benefits. It gives the character 2 conflicting goals: "Get the sister and prove that I'm worthy". Now, with those two we also get a kind of an achieving-plan: "If I'll show her that I'm worthy, by getting something amazing done, she'll want me and I'll be able to get her". More than that, the character gets the knowledge that each advancement in order to achieve one goal will drive the other one to the far end.
But the first conflict is even more interesting. The character here has the knowledge that he needs the brother in order to save his lover, but he just can't stand being near the brother. He'll drive the mission onward for two reasons but he'll have doubts about his lover- if he'll marry her he'll be stuck with this brother of hers.
To make long story short, simple conflicts can show the player that even normal characters are interesting and unique. When combined with goals they force the character to take certain steps along the roads, to commit certain actions along the way, that he won't want to do but will make him doubt himself and question himself and see that his problem are far more interesting than those of every freak that he'll encounter.
Another nice way to help him see the importance of conflict is through showing him and analyzing with him certain protagonists that are normal people, from the stories and movies and series (of any form)that he likes. He'll see quite quickly that the conflicts make them interesting.
But he may say that it is not enough. For that there are a few more literary tools that might help him see why normal people are interesting. The first one is having flaws (internal or external) and the second one is using "The Ghost".
Flawed characters are characters that just like normal people aren't perfect. Those flaws can be internal (self-doubts, for example, or a mild paranoia) or they can be external (they're look frightens ordinary people, for once, or a missing hand for the other). The idea is that the character has to deal with the flaw, and one day to find the strength to overcome it. The fight for the overcoming act makes the character far more interesting. A nice example of that can be seen in The Rain Man, where he learns at the end that he can count on strangers/"dumb" persons like he's brother. Another nice example can be seen in the story of The Ugly Duckling who although looking terrible learned to acknowledge himself and to accept the way he looks, to accept his difference.
"The Ghost" is an event from the past that just like a ghost haunts the character to this day. Again, trying to cope with it is what builds a deep character. One example for this can be seen in the movie Inception, where we literally have a ghost- Cob's wife. Another example for this can be seen in the movie Casablanca, where he has to deal with his broken relationship with Ilsa. This Ghost is far more interesting as the originator of the Ghost actually comes back to his life. In Frozen we see another kind of a Ghost- the act that one feels guilty about. Elsa actually killed her sister.
All of these techniques are there for one reason- to make regular people interesting, to give depth to the characters, to make them human beings with goals and drives and psychology.
Helping him see that his character doesn't fit the world
After he understands that he doesn't have to be a freak in order to be interesting, he will be far more understandable about playing a character that fits the world. Then, try to explain to him as calmly as you can what it is in the way he played his character that doesn’t fit the world.
Explain to him that the characters are in a world where being a freak is bad, where achieving one's goals is the ideal. Each and every one for himself, as the saying goes. Give him examples from the way he played his character and analyze with him, in a one-on-one conversation where his way of acting came from. Use the background he created to illustrate to him where your problem comes from.
Then ask him what problems he has with his character, and together try to find a solution. Maybe let him be just a little bit freakish. Maybe he needs to just create a different character. This is basically between you and him. After that show the updated character to the group and get their approval.
When combining those two, you'll get a player who his far more willing to both play the character while also seeing the problems with the way he played his character before.
Combining the two solutions
When combining the two solutions you get a better player, who understands for the future also how to create regular characters that are not freaks yet far more interesting than those freaks will ever be able to be. Furthermore, you get a player who is willing to play his character as written while still making the character fit into the world. Hope any of these helped you.
Best Answer
Deliberately taking disadvantage arguably starts to fall into "My Guy Territory", but simply showing fear and hesitation through roleplay would be a good thing at many tables.
Deliberately making your character underperform is likely to harm your entire group. Most groups will take a certain amount of umbrage to that if it is done knowingly. This is especially true when the way you make your character underperform is entirely mechanical.
What I mean by that is that there are some times when a choice that is theoretically suboptimal truly advances your character development. If for instance your paladin at low level and with poor equipment chooses to donate a large amount of money to the orphanage instead of buying better armor, that is clearly suboptimal from a purely combat perspective. But it does genuinely help with your character's characterization and development as a character beyond the numbers on the sheet and depending on the story and your DM might just pay dividends in some form later on. In contrast, asking for disadvantage is obviously suboptimal in a way that does not directly advance your character much or offer future story hooks. It is purely mechanical.
There might be some argument as to whether it is "My Guy Syndrome" as that is traditionally defined. But I think many groups would object to you doing it since it harms the whole group and the objection would be, in my humble opinion, reasonable.
On the other hand, you can add non-mechanical flourishes during combat to help with your character development. (I alluded to this in a previous semi-related answer about roleplaying fear).
Instead of asking for disadvantage, simply say "[Character] reluctantly attacks, pushing through his terror at this foe". Or "[Character's] heart is pounding as he confronts the situation, but knowing he must, he strikes", etc. The GM might possibly assign disadvantage, but at least then you aren't asking for it.
Later, when it changes, you can announce for characterization purposes that the fear and hesitation are gone and change your descriptions accordingly. There is no need for this change in characterization to affect the mechanics.