Here're the basics:
- Have the player sandbag. Explain to the player that his optimized PC makes DMing too difficult. The problem isn't that the player's winning—the DM, after all, has infinite monsters—but that the player's character is overshadowing the other players' characters. Strongly urge the player to pick a character class 1 to 2 tiers lower than the class of the highest or lowest tier PC.
- Have the player optimize other PCs. The player may be expecting other players to design characters of similar stature. However, other players may have different strengths and priorities beyond character optimization. But if he and the other players agree, it's reasonable to have the optimizer make other players' characters for them, putting them all on the same metaphorical level. The DM'll have to adjust challenges appropriately, however.
- Avoid conflict and suffer. Steel yourself to spending an inordinate amount of time customizing encounters for a single self-absorbed murder-machine and his party of lickspittles. During play, fudge die rolls to keep alive the lickspittles while still challenging the murder-machine. I advise against this, but it is an option. Good luck.
Now that those're out of the way.
Encourage the player to optimize his character for tasks other than monster-killing
Although this isn't an obvious path for a character to take in Dungeons and Dragons 3.5, such a character can be a interesting experiment when played by an experienced, squeeze-every-ounce-from-the-system player.
It's extremely easy to make a 3.5 character who can kill anything. It's much harder, for example, to make a character who can steal anything or heal anything. Explain to the player that everybody knows he can already play a character who can murder the campaign world and that playing that again wastes his talents. Have him turn his eye to little used subsystems, oft-ignored fringe mechanics, and seemingly-hard-to-optimize game elements.
Bear in mind, though, that he will use whatever he specializes in to try to solve every problem. Currently, this isn't that big of a deal—killing monsters solves a lot of problems—, but, if the specialty is extremely obscure, group cohesion and, subsequently, the campaign may suffer.
Mandate group cohesion
If the character is forced through mechanics or story to care about his fellow PCs, he should stop charging into rooms alone and murdering information-spouting NPCs.
Have him designate another PC as his brother, sister, childhood friend, lover, confessor, ward, or something, and tell him his character cares about that person deeply. If this a party of random murderhobos who met in a bar and decided to go camping forever, maybe the gods, fate, or prophecy binds them, and breaking that bond means Certain Doom.TM
The player needs a reason for his character to keep associating with these characters who are so obviously beneath him, and the player won't give his character this reason unless there's a mechanic that makes his character better because his character has that reason.1
Developing that reason, then, becomes the DM's job. The player probably won't like this mandate as it makes his character vulnerable in a way that mechanics can't usually minimize, but once the player starts engendering some goodwill from the other players by optimizing his character's ability to keep his friends alive and get along with the group in addition to his character's ability to kill faster (although, admittedly, that, too, can keep his friends alive), the player may have a more enjoyable experience playing that character than playing his typical brooding, uncaring lone wolf who only looks out for himself.
I GMed for the win-at-any-cost player for a decade. It's trying but does improve one's GMing skills. I've GMed the player—the same player—who thought the game would be better as a solo campaign and saw him come around to the idea of group cohesion after mandating it before the campaign began. Since I put forth that mandate in that campaign, I've done the same, to lesser and greater extents, in every campaign after. There's power to be had in saying A character who can't get along with the party isn't in the party and ending the conversation. Does saying…
- that the three months spent in a life raft created between the characters unbreakable bonds that will last the rest of the characters' lives, or
- that the characters' shared affection for their homeland despite their differing alignments unites them against common enemies, or
- that the royal family plucked the characters from obscurity to employ them as troubleshooters therefore the characters owe the royal family their unwavering service
…remove some agency? Sure. But, afterward, you can play the game. Role-playing games are (usually) cooperative affairs, and a player thinking his deliberately obstructionist, shenanigans-pulling character is acceptable makes the game a drag.
1 For this same reason, I strongly suspect all his characters but those needing to be otherwise are that least vulnerable of all the alignments, True Neutral.
This is fundamentally a question of playstyle
In differing styles players have jurisdiction over differing amounts and kinds of fictional material. In some styles, it would be completely inappropriate for you to determine any aspect of the PC's brother's character. In others, it would be completely inappropriate for the player to decide that his character even has a brother or any aspect of that character, should they exist.
Most of the time, there's some sort of balance between player and GM authority on most issues. As a matter of playstyle, groups might resort to the rules, out-of-game social hierarchies and methods of discrimination, shouting matches, open discussion and compromise, or any manner of other methods to determine the nature of the game world when there is a dispute of some kind.
Your asking this question makes it clear that your group hasn't yet formed a position within this question's scope for the purposes of your game. That's not a bad thing-- playstyle is often most meaningfully developed during play. That said, you have correctly identified this as an area of concern: your player has invested a lot in their PC's attachment to the brother, and if it goes 'wrong' people may well be upset. The solution here is to talk about what you guys want as a group regarding fictional authority over PC backstory relationships as well as PC backstory relations. You don't need to specifically voice your plans with the group regarding the brother turning out to be the BBEG or whatever you want to later be a reveal; discuss the stance of the group in a general sense, for example by asking "Is it okay for me to keep information about important characters secret from you in order to reveal it later" and "How much control over your brother should I have". Ultimately, figuring out how you want to deal with this will be a formative process for your group.
The Problem: There's a Lot of Stuff in a Backstory
In my experience, mining ideas from PCs' backgrounds can be a bit tricky. That's because a relationship or event written into a backstory can serve one of several quite distinct purposes. For example:
At times, trying to divine what exactly a player hopes to do with a particular bit of a backstory can be tricky. In my experience, there are two particular issues to watch out for:
In many games, a PC's background is the main way in which non-GM players create details about the world. Bringing that stuff into play is good, but it's possible a player will feel disempowered when you start putting your own spin on their relationship characters or make their backstory part of a plot twist.
Character backstories are often about defining a personality. Sometimes bringing in an element too directly means some fundamental character-defining conflict is resolved, and a player feels like their PC's arc as a protagonist has concluded prematurely.
Luckily, it's easy to avoid these missteps with a bit of communication.
A Solution: Highlight What You Want to See
How do you know which elements of a character's background their player really wants to bring up in play? Just ask.
Some games use game mechanics to guide these discussion; these kinds of mechanic are called "flags" (most clearly discussed here: practical examples of good flags, some technical details). You can add a basic version of this framework to existing games pretty easily. Here's one example of a quick-and-dirty method.
For backstories specifically, ask players to highlight what they want to see reincorporated in play.
For example, ask everyone to write down, separate from any long-form backstory:
You can look at these sheets during planning or in play to help guide your GMing. Encourage players to modify them over time.
How Do I Make This "Seamless?"
If you're using the method above, look at these notes during planning and play. Make it a part of session planning, encounter design, or scene framing. Incorporate designated major backstory characters into your own relationship maps, side-by-side with the NPCs you've invented yourself.