[RPG] What to do when your character is just too good

Here're the basics:

1. Have the player sandbag. Explain to the player that his optimized PC makes DMing too difficult. The problem isn't that the player's winning—the DM, after all, has infinite monsters—but that the player's character is overshadowing the other players' characters. Strongly urge the player to pick a character class 1 to 2 tiers lower than the class of the highest or lowest tier PC.
2. Have the player optimize other PCs. The player may be expecting other players to design characters of similar stature. However, other players may have different strengths and priorities beyond character optimization. But if he and the other players agree, it's reasonable to have the optimizer make other players' characters for them, putting them all on the same metaphorical level. The DM'll have to adjust challenges appropriately, however.
3. Avoid conflict and suffer. Steel yourself to spending an inordinate amount of time customizing encounters for a single self-absorbed murder-machine and his party of lickspittles. During play, fudge die rolls to keep alive the lickspittles while still challenging the murder-machine. I advise against this, but it is an option. Good luck.

Now that those're out of the way.

Encourage the player to optimize his character for tasks other than monster-killing

Although this isn't an obvious path for a character to take in Dungeons and Dragons 3.5, such a character can be a interesting experiment when played by an experienced, squeeze-every-ounce-from-the-system player.

It's extremely easy to make a 3.5 character who can kill anything. It's much harder, for example, to make a character who can steal anything or heal anything. Explain to the player that everybody knows he can already play a character who can murder the campaign world and that playing that again wastes his talents. Have him turn his eye to little used subsystems, oft-ignored fringe mechanics, and seemingly-hard-to-optimize game elements.

Bear in mind, though, that he will use whatever he specializes in to try to solve every problem. Currently, this isn't that big of a deal—killing monsters solves a lot of problems—, but, if the specialty is extremely obscure, group cohesion and, subsequently, the campaign may suffer.

Mandate group cohesion

If the character is forced through mechanics or story to care about his fellow PCs, he should stop charging into rooms alone and murdering information-spouting NPCs.

Have him designate another PC as his brother, sister, childhood friend, lover, confessor, ward, or something, and tell him his character cares about that person deeply. If this a party of random murderhobos who met in a bar and decided to go camping forever, maybe the gods, fate, or prophecy binds them, and breaking that bond means Certain Doom.^TM

The player needs a reason for his character to keep associating with these characters who are so obviously beneath him, and the player won't give his character this reason unless there's a mechanic that makes his character better because his character has that reason.¹

Developing that reason, then, becomes the DM's job. The player probably won't like this mandate as it makes his character vulnerable in a way that mechanics can't usually minimize, but once the player starts engendering some goodwill from the other players by optimizing his character's ability to keep his friends alive and get along with the group in addition to his character's ability to kill faster (although, admittedly, that, too, can keep his friends alive), the player may have a more enjoyable experience playing that character than playing his typical brooding, uncaring lone wolf who only looks out for himself.

Note

I GMed for the win-at-any-cost player for a decade. It's trying but does improve one's GMing skills. I've GMed the player—the same player—who thought the game would be better as a solo campaign and saw him come around to the idea of group cohesion after mandating it before the campaign began. Since I put forth that mandate in that campaign, I've done the same, to lesser and greater extents, in every campaign after. There's power to be had in saying A character who can't get along with the party isn't in the party and ending the conversation. Does saying…

• that the three months spent in a life raft created between the characters unbreakable bonds that will last the rest of the characters' lives, or
• that the characters' shared affection for their homeland despite their differing alignments unites them against common enemies, or
• that the royal family plucked the characters from obscurity to employ them as troubleshooters therefore the characters owe the royal family their unwavering service

…remove some agency? Sure. But, afterward, you can play the game. Role-playing games are (usually) cooperative affairs, and a player thinking his deliberately obstructionist, shenanigans-pulling character is acceptable makes the game a drag.

¹ For this same reason, I strongly suspect all his characters but those needing to be otherwise are that least vulnerable of all the alignments, True Neutral.

Metagaming

First, lets kill the metagaming ad hominem: "Metagaming is any strategy, action or method used in a game which transcends a prescribed ruleset, uses external factors to affect the game, or goes beyond the supposed limits or environment set by the game. Another definition refers to the game universe outside of the game itself."

Optimising a combat-oriented character to be good at combat within the rules is not and never can be metagaming. You can't even mount a game universe argument that it is: a person who has devoted his life to being a wilderness warrior (aka a Ranger) is going to learn to be good at fighting or die!

Comparison

Hit Points

Your hit point edge is insignificant; an 11 hp advantage is, on average, 2 hits or 1-2 rounds more staying power in a combat (less if fighting multiple foes). When you consider that the Paladin has an ability to heal 15 hp with their Lay on Hands ability at the cost of an action, they effectively have more hp than you do. You do have a definite advantage if you are being hit by things like fireballs; on failed saves you are the only one left standing.

This is an edge but a small one.

Damage output

I will assume everyone has the same stat modifier on damage rolls.

If you are using your bow and choose to use a spell slot for Hunters Mark, you can do 3 + d8 (bow) + d6 (Hunters Mark) (avg 11) on the first hit and the same plus d8 (Colossus Slayer) (avg 15.5) on subsequent attacks. This is great if you are fighting a monster with lots of hit points; it is not so good against a dozen goblins since the first hit will drop them and your Colossus Slayer never kicks in.

Meanwhile the Paladin with a longsword and the dueling fighting style is doing 3 + 2 + d8 (longsword) + 2d8 (Divine Smite) (avg 18.5) (I haven't considered some of the really cool spells they have).

The Rogue is doing 3 + d8 (longbow) + 2d6 (sneak attack - a good rogue should almost always get this) (avg 14.5).

The Sorcerer has a plethora of options (Magic Missile, Burning Hands, and Cloud of Daggers spring to mind) or they can just fall back on a damaging cantrip for d10 (avg 5.5). If they are a gambler, Hold Person can end a combat with a single humanoid on one failed saving throw.

If the Bard wants to be handing out massive damage in combat then they chose the wrong class; that is not where their talents lie, they are an enabler - they enable others to do more damage.

The Ranger is not the best at handing out damage.

Overall, you are playing your character to his strengths; are the other players playing to theirs?

Pacing and Encounter structure

You say "I was typically able to go first in any combat due to high DEX, and dealt such insane damage that the guys going last did nothing".
I read "The encounters are underpowered".

Don't misunderstand me: it is the nature of RPG that the PCs will win (almost) every fight because they can only lose once. Most combats will be and should be cakewalks, they are there because combat is fun and they consume resources. That said, they shouldn't be so insignificant that they are over before the first round ends. A quick combat like this is great if the players have planned and executed a great ambush, its not great if it is just way underpowered.

If you have enough spells to use a spell in every combat then you are not having enough encounters between long rests. Burning through spell slots for a non-core spellcaster should be a tough decision: "Do I use it now or will I need it latter?" If you are not thinking this, at least briefly, all the time then your DM is being easy on you. Fights early in the day will usually be easy but this is due to everyone having lots of resources, as you burn through spell slots and hp the same encounter becomes much harder.

Also, the structure of encounters matters. 5 PCs on one monster is an easy fight (unless the monster's CR is extremely high for the party); the monster can only target 1 PC while copping damage from all 5. 5 PCs on 5 monsters is much harder; the tough PCs have to control the battlefield or the squishy PCs will get squished. 5 PCs on 15 monsters, even very weak monsters, is really hard; everyone is copping damage and the fight will last 4-5 rounds minimum.