[RPG] How I politely say to the players that I don’t want to DM to them anymore

You're not having fun.

Since the entire point of role-playing is to have fun, you need to do something about it. In particular, it sounds like the character he's playing (called a GMPC) is seriously distracting him from his GM task of running a fun story for the other players at the table.

You have several options, depending on whether you're willing to accept him as a GM as long as he changes, or if you can accept him as a part of the group as long as he isn't GMing:

1. Talk to the GM about his behavior. Try to be non-confrontational, and don't have this conversation in front of the group; that will just make him defensive and won't accomplish anything. Try persuading him to not bring a character along with the group; he's GMing, not playing.
2. Sit out the current game, until the GM is done. There's no need for you to continue playing a game you dislike, so take a break until he's done running the game. Then join the next game, that someone else in the group will run.
3. Take over GMing duties, with the support of the rest of the group. Since they already asked you to be the GM, they'll almost certainly be willing to let you run the game. Talk to the other players about this individually before you try this; if you have enough support, you'll be able to usurp his position as GM, and start an entirely different game. Perhaps the current GM will be a better player than they are a GM. And look on the bright side: you now have some fantastic examples of how not to GM, which will make you much better at this new role! You can also combine this with #2 above: sit out the game until it's over, then be the next GM. (Warning: given what you've described of his personality, the current GM is unlikely to take this well.)
4. Start a separate gaming group, inviting whoever is willing to play without the problem GM. If some people are unwilling to play with the new group because they don't want to exclude the current GM, then you'll need to exclude them as well. Recruit a few other players from elsewhere if you need to.
5. Find a new gaming group. If all else fails, just don't play with this group at all. If they're willing to play with someone they dislike, and you can't persuade them to kick out a problem GM, that's their problem. It doesn't have to be your problem.

Note that none of these options include playing a game you dislike with a GM who you hate. He isn't going to improve on his own, unless you talk him into changing or encourage the group to leave him behind.

On a slight tangent, see the Geek Social Fallacies. It sounds like some of the members of the gaming group are suffering from GSF #1: "Don't exclude anyone" and GSF #5: "Failing to invite someone is a deliberate snub."

I do several things to keep the player characters interested and invested in each other.

1. At the start of the game, I insist that players coordinate backgrounds (subject to my approval) such that each character know at least one, and preferably two or more, of the other characters. In general, I prefer these connections be positive; the most negative I will usually tolerate is on the order of a friendly rivalry. In general, I also prefer that if you start from any one character, you can get to any other character by following these pre-arranged links. (In graph theory language, the players are all connected, although indirect connections are fine; the alternative would be two more more sub-groups connected internally but not to each other.) And finally, I try to ensure that each connection is more than trivial, but not necessarily life-binding.

So for instance, "We met in a bar twenty years ago and never saw each other again," is trivial. "We are cousins who are best friends and we are rarely separated," is life-binding and more than I look for (although it's fine if that's what they want.) Things like the following are what I look for, and/or what I've seen in the past:

• Our characters served in the same unit years ago, and knew each other, but haven't kept in contact...
• I served as a mercenary escort once, while he was travelling with his master from here to there; along the way, this happened....
• We weathered the siege/plague/earthquake of wherever together some time back....

Now, some players are genuinely not wired that way-- if you ask them for backstory, they freeze; if you given the one, they can't connect to it. When I run into a player like that, I have to respect that, but I try very hard to get everyone to adhere to the guidelines.

That does not directly solve the problem. (It actually solves the problem of getting the characters all on the same page at the start of the game.) But it does often give me enough to work with to do the following:

2. With enough insight into character backgrounds, and with overlapping backgrounds, I try to give every character an mid-term to long-term goal or plot arc, and then I try to modulate that by giving at least one other character a minor to moderate interest in how the first character's arc plays out.

It's important (to me, for the games I want to run) that these arcs not be strictly opposing: If one character has sworn blood-vengeance on an NPC, I won't give another character the goal of keeping that NPC alive. But I might give another character the goal of getting something from that NPC before his death, or getting the NPC to do something, etc.

3. And I also try to modulate this in another direction by giving other characters-- ideally, not the same one-- influence over the plot lines. So continuing that thought:

   • Player A has sworn to kill Sir Odious, his parents' killer
   • Sir Odious has information that will help Player B in her quest to do something else
   • Player C knows someone who can be bribed into giving up information about where Sir Odious will be

In that way, for each of the various player sub-quests going on, at least one or two others will be involved somehow, even if only at the periphery. Ideally, Player B has some motivation for something to happen, and Player C has something he needs-- something at least moderately costly or risky. They are invested.

One thing I would not do-- at least not again-- is what you tried:

I've had players create characters (with backgrounds) completely secretly from each other with the hopes of allowing the character interaction to be heavily role-played at the table. Didn't work because of very incompatible characters.

I've never done that, specifically, but I've inadvertently done similar things and it never worked well. It seems like it should work, especially if you pattern it similar to what I've outlined above, but there's a structural weakness to it: If the players, starting out with the relative blindness of only knowing their little part of the background, they just might not see those connections you built in for them, and won't give themselves the incentive to start sharing information. And if your players were the sort that would do that naturally, you wouldn't have to go through these acrobatics in the first place.