To make your question short, and to see if I understood it correctly, we're talking about a player who made his character a certain one and roleplays it entirely different. You added that you think that it comes from inexperience, and that he created this character after you said "no" to some "freak-character"-ideas. You want to help him roleplay the character he created.
As I see it, this problem is made from two smaller ones. The first is that he doesn't see his character as interesting because the character "is normal and normal is boring". The second is that you wanna help him understand why the way he plays the character does not fit the story-world of your game.
Helping him understand that "normal is not boring
This is the more important problem, as it stands in the basis of the entire problem. If he'll see that normal characters can be interesting his "anti-persona" will perish and he'll roleplay a normal character and not a freak one. The main trick here is to show him that normal characters are not entirely normal, i.e. "no person is like the others". In order for that to work, we need to give the character depth.
The easiest way to give depth to a character is through internal conflicts. Having goals and all is nice, but without something that blocks oneself from achieving them it is far less interesting. First thing to do is to go over his character's background and see if he implemented there an internal conflict for his character. If so, show it to him and talk with him about it. If no, sit with him and help him to come with one. The internal conflict doesn't have to be extravagant, but it needs to be there. An example one might be that he loves Vincent's sister but secretly hates Vincent himself, or another like Loves the sister but thinks that he's not good enough for there. I'll take the second one as an example for this section.
The conflict gives us a few things, a few added benefits. It gives the character 2 conflicting goals: "Get the sister and prove that I'm worthy". Now, with those two we also get a kind of an achieving-plan: "If I'll show her that I'm worthy, by getting something amazing done, she'll want me and I'll be able to get her". More than that, the character gets the knowledge that each advancement in order to achieve one goal will drive the other one to the far end.
But the first conflict is even more interesting. The character here has the knowledge that he needs the brother in order to save his lover, but he just can't stand being near the brother. He'll drive the mission onward for two reasons but he'll have doubts about his lover- if he'll marry her he'll be stuck with this brother of hers.
To make long story short, simple conflicts can show the player that even normal characters are interesting and unique. When combined with goals they force the character to take certain steps along the roads, to commit certain actions along the way, that he won't want to do but will make him doubt himself and question himself and see that his problem are far more interesting than those of every freak that he'll encounter.
Another nice way to help him see the importance of conflict is through showing him and analyzing with him certain protagonists that are normal people, from the stories and movies and series (of any form)that he likes. He'll see quite quickly that the conflicts make them interesting.
But he may say that it is not enough. For that there are a few more literary tools that might help him see why normal people are interesting. The first one is having flaws (internal or external) and the second one is using "The Ghost".
Flawed characters are characters that just like normal people aren't perfect. Those flaws can be internal (self-doubts, for example, or a mild paranoia) or they can be external (they're look frightens ordinary people, for once, or a missing hand for the other). The idea is that the character has to deal with the flaw, and one day to find the strength to overcome it. The fight for the overcoming act makes the character far more interesting. A nice example of that can be seen in The Rain Man, where he learns at the end that he can count on strangers/"dumb" persons like he's brother. Another nice example can be seen in the story of The Ugly Duckling who although looking terrible learned to acknowledge himself and to accept the way he looks, to accept his difference.
"The Ghost" is an event from the past that just like a ghost haunts the character to this day. Again, trying to cope with it is what builds a deep character. One example for this can be seen in the movie Inception, where we literally have a ghost- Cob's wife. Another example for this can be seen in the movie Casablanca, where he has to deal with his broken relationship with Ilsa. This Ghost is far more interesting as the originator of the Ghost actually comes back to his life. In Frozen we see another kind of a Ghost- the act that one feels guilty about. Elsa actually killed her sister.
All of these techniques are there for one reason- to make regular people interesting, to give depth to the characters, to make them human beings with goals and drives and psychology.
Helping him see that his character doesn't fit the world
After he understands that he doesn't have to be a freak in order to be interesting, he will be far more understandable about playing a character that fits the world. Then, try to explain to him as calmly as you can what it is in the way he played his character that doesn’t fit the world.
Explain to him that the characters are in a world where being a freak is bad, where achieving one's goals is the ideal. Each and every one for himself, as the saying goes. Give him examples from the way he played his character and analyze with him, in a one-on-one conversation where his way of acting came from. Use the background he created to illustrate to him where your problem comes from.
Then ask him what problems he has with his character, and together try to find a solution. Maybe let him be just a little bit freakish. Maybe he needs to just create a different character. This is basically between you and him. After that show the updated character to the group and get their approval.
When combining those two, you'll get a player who his far more willing to both play the character while also seeing the problems with the way he played his character before.
Combining the two solutions
When combining the two solutions you get a better player, who understands for the future also how to create regular characters that are not freaks yet far more interesting than those freaks will ever be able to be. Furthermore, you get a player who is willing to play his character as written while still making the character fit into the world. Hope any of these helped you.
1. Get a cheatsheet into each player's hands.
You know that godsend player, the one who always has the notecards? Key thing there: the notecards.
You've spoken to the group, and they got upset, but you know they cared enough to get the books in the first place. It's entirely possible that they do just forget, or maybe they're having a difficult time with the rules and don't know how to articulate it. Even if they read cover to cover (and maybe they already have), there's a lot of information to parse and remember, especially for new players. Even veterans forget rules sometimes! Still, fumbling through the book for basic rules is not time-effective. My solution is to create a cheatsheet. This has helped me as both GM and player.
Figure out your most commonly used rules. Type up these rules and print a copy for each player. Try to condense them as much as you can. Strip them to their bare bones so players can find the information they need at a glance. Also, since you'll be distributing the same page to everyone, concentrate on rules that everyone uses. I'll discuss character-specific rules later on.
It's just more practical for you to make the cheatsheet. If they don't have the rules down yet, they probably don't know which ones are the most important. You likely have a better grasp, and you're also the one who wants more efficiency. The best way to ensure that this happens is to do it yourself, and then the whole group will be on the same page.
For character-specific rules, though, it's best if the players do it. This spreads the effort around. Plus, you can use this as an opportunity to help re-familiarize your players with their mechanics. Sit down with them and help them make index cards or type notes from the book. Point out nifty features and answer any questions.
Assure them that it will make their lives way easier, because it will. They probably don't like book-fumbling either. They'll also be more likely to use their character's features if they're always right in front of them in an easily-digestible format. It's hard to achieve your full potential if you don't know where to find it. The key thing is to emphasize how useful it will be for them (in character and out) if they do this, not how much worse things will be if they don't. Make it about them and their potential fun, not your frustration, to avoid further souring.
It's awesome that the Battlemaster Fighter took the initiative, though. If this is something he enjoys doing, you could enlist his help! I'm this player in my own groups, even when I'm not GMing. I love having a cheat sheet, so if I'm making one anyway, why not share? At the least, use his notes as an example (if he's okay with it).
2. Make use of existing resources.
Cheatsheets you make will be the best-tailored to your needs. That doesn't mean you shouldn't use ready-made tools on top of that. People often share their own tools if they think they're especially useful. If you need something specific, search the web to see if it already exists. There's no guarantee that what you find will be useful -- again, you know your own needs best -- but it never hurts to try. If you hit gold, that's time and effort saved.
For instance, here are some form-fillable initiative cards that provide a quick reference of initiative order and PC skills, traits, and actions; here is a larger, longer version of a similar concept. Here's a site that generates spell cards. ENworld has many sheets available. It only took me a few minutes to locate them by searching "5e spell cards," "5e initiative cards," and "5e character sheets," respectively. You'll be more successful if you already know what you need, but you can still try searches like "5e game resources" and "5e player print outs" if you're looking for tools in general.
Best Answer
He doesn't need to pick up the books in the first place. RPG's are a recreational activity. If he doesn't want to spend time reading gaming books, then he really has no obligation to do so. Maybe tabletop RPG's just aren't for him. Perhaps he's just going along with everyone because he wants to hang out with the group or doesn't want to be the wet blanket on the group's activity, but otherwise isn't very interested.
I wouldn't be too concerned about it. If he tries gaming and enjoys it, he'll start reading the books and becoming familiar with the system out of new found interest. If he doesn't enjoy it, then you'll probably lose him as a player anyways. But badgering him to do 'homework' is most assuredly not going to help you keep him around.