To make your question short, and to see if I understood it correctly, we're talking about a player who made his character a certain one and roleplays it entirely different. You added that you think that it comes from inexperience, and that he created this character after you said "no" to some "freak-character"-ideas. You want to help him roleplay the character he created.
As I see it, this problem is made from two smaller ones. The first is that he doesn't see his character as interesting because the character "is normal and normal is boring". The second is that you wanna help him understand why the way he plays the character does not fit the story-world of your game.
Helping him understand that "normal is not boring
This is the more important problem, as it stands in the basis of the entire problem. If he'll see that normal characters can be interesting his "anti-persona" will perish and he'll roleplay a normal character and not a freak one. The main trick here is to show him that normal characters are not entirely normal, i.e. "no person is like the others". In order for that to work, we need to give the character depth.
The easiest way to give depth to a character is through internal conflicts. Having goals and all is nice, but without something that blocks oneself from achieving them it is far less interesting. First thing to do is to go over his character's background and see if he implemented there an internal conflict for his character. If so, show it to him and talk with him about it. If no, sit with him and help him to come with one. The internal conflict doesn't have to be extravagant, but it needs to be there. An example one might be that he loves Vincent's sister but secretly hates Vincent himself, or another like Loves the sister but thinks that he's not good enough for there. I'll take the second one as an example for this section.
The conflict gives us a few things, a few added benefits. It gives the character 2 conflicting goals: "Get the sister and prove that I'm worthy". Now, with those two we also get a kind of an achieving-plan: "If I'll show her that I'm worthy, by getting something amazing done, she'll want me and I'll be able to get her". More than that, the character gets the knowledge that each advancement in order to achieve one goal will drive the other one to the far end.
But the first conflict is even more interesting. The character here has the knowledge that he needs the brother in order to save his lover, but he just can't stand being near the brother. He'll drive the mission onward for two reasons but he'll have doubts about his lover- if he'll marry her he'll be stuck with this brother of hers.
To make long story short, simple conflicts can show the player that even normal characters are interesting and unique. When combined with goals they force the character to take certain steps along the roads, to commit certain actions along the way, that he won't want to do but will make him doubt himself and question himself and see that his problem are far more interesting than those of every freak that he'll encounter.
Another nice way to help him see the importance of conflict is through showing him and analyzing with him certain protagonists that are normal people, from the stories and movies and series (of any form)that he likes. He'll see quite quickly that the conflicts make them interesting.
But he may say that it is not enough. For that there are a few more literary tools that might help him see why normal people are interesting. The first one is having flaws (internal or external) and the second one is using "The Ghost".
Flawed characters are characters that just like normal people aren't perfect. Those flaws can be internal (self-doubts, for example, or a mild paranoia) or they can be external (they're look frightens ordinary people, for once, or a missing hand for the other). The idea is that the character has to deal with the flaw, and one day to find the strength to overcome it. The fight for the overcoming act makes the character far more interesting. A nice example of that can be seen in The Rain Man, where he learns at the end that he can count on strangers/"dumb" persons like he's brother. Another nice example can be seen in the story of The Ugly Duckling who although looking terrible learned to acknowledge himself and to accept the way he looks, to accept his difference.
"The Ghost" is an event from the past that just like a ghost haunts the character to this day. Again, trying to cope with it is what builds a deep character. One example for this can be seen in the movie Inception, where we literally have a ghost- Cob's wife. Another example for this can be seen in the movie Casablanca, where he has to deal with his broken relationship with Ilsa. This Ghost is far more interesting as the originator of the Ghost actually comes back to his life. In Frozen we see another kind of a Ghost- the act that one feels guilty about. Elsa actually killed her sister.
All of these techniques are there for one reason- to make regular people interesting, to give depth to the characters, to make them human beings with goals and drives and psychology.
Helping him see that his character doesn't fit the world
After he understands that he doesn't have to be a freak in order to be interesting, he will be far more understandable about playing a character that fits the world. Then, try to explain to him as calmly as you can what it is in the way he played his character that doesn’t fit the world.
Explain to him that the characters are in a world where being a freak is bad, where achieving one's goals is the ideal. Each and every one for himself, as the saying goes. Give him examples from the way he played his character and analyze with him, in a one-on-one conversation where his way of acting came from. Use the background he created to illustrate to him where your problem comes from.
Then ask him what problems he has with his character, and together try to find a solution. Maybe let him be just a little bit freakish. Maybe he needs to just create a different character. This is basically between you and him. After that show the updated character to the group and get their approval.
When combining those two, you'll get a player who his far more willing to both play the character while also seeing the problems with the way he played his character before.
Combining the two solutions
When combining the two solutions you get a better player, who understands for the future also how to create regular characters that are not freaks yet far more interesting than those freaks will ever be able to be. Furthermore, you get a player who is willing to play his character as written while still making the character fit into the world. Hope any of these helped you.
I'm sorry to tell you this but I don't think the RPG stackexchange is going to be able to help you a lot with this. The issue is really specific and requires the solution to be tailored just for him. If you need help on the matter I suggest you ask a psychologist for tips.
I can make a few suggestions on how to deal with the situation but please note that I have zero experience with dealing with this kind of issue, opposed by your 10 years. You'll have to decide for yourself, using your own experience and wishes if these suggestions are worth considering. And on a sidenote, thumbs up for being a good friend to him.
Play with his friends
If playing with strangers makes him uncomfortable then try and play with friendly faces. These people will (hopefully) know his issues and will be less mad at him for causes disruptions. You've learned to deal with your friend and learned to be around him. Strangers have not and might not be willing to do so.
Of course you can only play with friends if you have enough of friends that are interested in tabletop gaming.
Play solo campaigns, for now.
It is very much possible and enjoyable to play games with just one player. This might seem weird at first but it might be worth something considering. The main purpose of this is to keep him in his comfort zone. If it's just you he will have no reason to be nervous. If you do this for a while you might consider bringing in more players, one at a time. Just try to keep him in his comfort zone.
The problem with this solution is that your friend might consider this 'giving up'. My guess is that he knows rejection by now and will just consider this giving in to the rejection from other players. The point of this idea is to build up slowly and to work around his social anxiety that way, not to isolate him. So if you are going to take this route make sure to try and tell it him in a positive way. You'll have to use your knowledge about your friend to decide how to do this for yourself.
Also, solo campaigns take a lot of time to prepare.
Online gaming
This solution isn't great, but's it there. You and your friend could try and play in text based RPGs. Of course this is something that you need to enjoy and isn't a solve-all solution, it's more of a workaround. But if you both enjoy it them it might be something to consider.
Ask him for suggestions and feedback.
If anyone knows what the problems are, it's probably your friend. You've said this yourself in your post, but it's important enough that I'll repeat it here.
If you find any possible solutions or workarounds for said problems, talk them with through your friend. I know I'm just stating the obvious here but really is that important.
You might also want to ask him if he has any solutions for the problem if you haven't already (But I'm kind of assuming you already have.).
In short.
The gist of these suggestions is to work around his social anxiety as he obviously (and understandably giving his condition) has a lot of issues with it. Try to get him in a comfortable spot and hopefully his ticks will he less frequent. Make sure that you pick other players who are okay with playing with him.
Best Answer
Have dinner with your friend and help her draw up a checklist.
This advise is taken from my own personal experience doing exactly this. And it did help the person I was helping significantly, though as part if it, we also created a character to suit their requirements.
Checklists are an amazing technology that are astonishingly underrated by many people. Functionally, the checklist's role is to enable pre-cognition outside the heat of the moment. With any sort of complexity problems, a checklist is a good way to reduce the granularity of the problem such that a player can concentrate on the novel elements.
So, while I made a checklist for my epic character, with a list of the actions I would take if this or that were true, or on the first turn, the point of the checklist is not to strictly adhere to the thing, but to reduce the need to think about all of the things on the list.
When having this conversation, first discuss common "patterns" in combat. Figure out common situations that she remembers from game. While there are plenty of theoretical situations, it's more important when doing this to let the cognition and inspiration of tactics come from the individual. Just like with making a cheat sheet, the act of making the thing is more important than the thing itself.
After listing all of the common situations that she can remember, work through sample outcomes with her in whatever way suits you two best. (My preference is for statistical analysis, but then I'm very special. One of the simpler ways is to play it through a few times, alternating sides and talking through your decision.)
After each finishes, let her work out a checklist of "if this situation is happening try for X,Y,Z" Then, after a set of checklists are done, let her employ them in game. After she does, make sure to debrief her after game (written or oral, since there is some literature that suggests a written debrief has great learning potential) and talk through what went right, what went wrong, and what needs to change in the lists.