[RPG] Is it plausible to have a Player Character play as a villain against the rest of the party

I've run a variety of tones of campaigns over time and some could be considered "evil"; in fact currently I'm running a three-year long Pathfinder campaign where the PCs are pirates - not all of them are technically evilly aligned, but murder, torture, rape, slavery, etc. have all come up in the game. Here's how you make it work.

Decide on Limits, Within Limits

Some people, when they say "evil campaign," just mean "I want to kill lippy villagers like they're orcs," not that they want to really delve into the darker aspects of human nature. Establish an agreement on tone/content with your players up front - you are not required to run (and the players aren't required to participate) in anything they feel like is over their boundaries. I've been known to have players vote on approximate levels of sex, violence, etc. in a game ahead of time, and where they want it to "fade to black."

However, a lot of that will be emergent. In my current pirate campaign, no one really thought about torture until they caught an assassin who was trying to kill the crime-boss they were aligned with. The PC halfling rogue decided he'd torture her extensively to find out who sent her. This definitely put off the other PCs - but not enough that they stopped him. Boundary established (well, lack of one).

Not every "evil" person is 100% evil and on board with everything "evil," though. The ship took two elven women prisoner and one was claimed as a slave by a vicious half-orc pirate. The PC captain didn't really like that but felt somewhat constrained by the expectations of the crew (mutiny is always a threat if the crew doesn't think they're getting their due) so he allowed it. The PCs and that half-orc were having dinner in the captain's cabin, and the halfing from the anecdote above suddenly stabbed the half-orc to death on the dinner table (he's an assassin now - successful death attack). He explained to the shocked command staff that he wouldn't have any slaves on board or associate with slavers. Boundary established.

If you have real characters really roleplaying and thinking through their motivations, you'll still have limits, whether it's "no women, no kids" or the Mafioso that are patriotic and still want neighborhoods to be "family places." Try to depict other "evil" people as complex in that way as well so that they will understand that evil isn't just a race to maximum depravity. With that halfling, torture of captives is OK but slavery and rape is a killin' offense. There's no "Evil Checklist" you have to adhere to and say every crime ever considered is OK - in fact most evil people really are just into one and consider the others to be as bad as other folks do.

However - people make too much of setting boundaries for their games sometimes. If you came up to me and asked me "Do you want to see some chick saw her cheeks off?" I'd say "No! What are you talking about?" But I just went to see the movie Evil Dead, where that exact thing happened as part of the overall horror movie experience. "Boundary pushing" can be good and desirable and allowed based on initial buyin to the general campaign premise. Sure, there's a very slight majority of people so traumatized by something that if it comes up in game it's going to truly trip them out, and there you have outs just like any other kind of media - "press stop," say "I can't deal with this" - but most gaming groups don't really need to do more than establish the general MPAA-rating (e.g. "Hey guys I'm active in my church and I don't really want to go past PG-13 with this game") and then mess around in that area. Other questions here about "oh how do I double check with my players some specific thing is OK before doing it" is overthinking it IMO. If you go see Evil Dead, you'd better expect that if you have a fear of/complex about anything, there's a nonzero chance it's going to come up in lurid color. All the buyin we required for the pirates game was "people can be evil if they want, and expect HBO Original Series level depravity, the pirate world is not a gentle one."

But What's It Really About?

"Evil" is not really a campaign concept (well, not one that passes muster past the 9th grade level). You need a campaign concept and one that will generally keep the PCs acting together instead of being at each others' throats (unless you're looking for a more short, PvP campaign, which is legitimate and there's plenty of short form indie games that facilitate that). If you are more going for "longer D&D campaign," which I'm reading between the lines of your question that you are, it needs to have as much goal as any other campaign. Smart PCs know they need other mighty people to achieve their goals.

Heck most "normal" campaign setups work as well or better with evil groups - just because you're evil, you don't really want where you live and work taken over by zombies or whatever, that interferes with your cashflow. Often times players want to "play evil" because they feel like the GM has been using "goodness" to manipulate them into being passive and they want to be proactive and smart in confronting threats. Squinting too hard at many campaign concepts passed off as "good" reveals them to be a sequence of home invasion, murder, and robbery anyway.

The main trap you're trying to avoid is the PCs just self destructing by going nuts on each other and everyone in the world in general - at least, if they'd be unhappy with being hunted down and slain a couple sessions in.

Actions Have Consequences

Review How do I get my PCs to not be a bunch of murderous cretins? - there are a lot of reasons people don't perform unrestrained evil deeds all the time, from "I don't want to" to "I will get in trouble for it." Sometimes my players complain that the pirate-friendly port city they frequent is "too lawful" just because they can't get away with any heinous crime or breach of the peace they can come up with - but all societies need some kind of stability and will crack down on those affecting that too much. On the other hand, they have become used to not going out into the city alone; traveling in groups is mandatory to not be victimized themselves.

Many evil societies are like this - see how lawful Drow society looks from the outside. Our pirate PCs have to fear their pirates mutinying, the law/navy hunting them down, the bigger pirates in port deciding they're too big for their britches or have so much loot that they're a tempting target in turn. Criminals "hide out" for a reason - they are not free to operate within larger society, and therefore end up having less freedom than good people (something good to play up as the GM). The law, higher level "good" adventurers, etc. are always looking to wipe you out with a clear conscience.

A mechanical option here is keeping track of "infamy points" - I have my own homebrew system I use, but there's a lot of extant reputation-tracking mechanics in the world. People have heard of the big bad people and will react like people do - avoid, confront, narc them out, victimize them, etc. Remember that many victims of crime are doing something bad themselves - criminals, or at least the dishonest, make the best marks for cons and crimes because they have little legal recourse. The pirate PCs can't go just anywhere as their infamy becomes known; honest ports reject them, and other evil folks are generally not the best allies because they like to turn on you when you blink.

Why Do It?

There's a couple reasons to run an evil campaign and the measure of success is different per type.

1. I want to freak out and kill everyone! Not a real mature campaign and you should probably decline. Tell them "go play Call of Duty and teabag noobs if that's what you want." There is no meaningful success metric here.
2. I want freedom! As I mentioned before, much of the time people want an 'evil campaign' it's because they feel constrained/manipulated by their GM and/or other players based on an overly restrictive interpretation of alignment. In that case I'd run an evil campaign once, use it to demonstrate that criminals generally enjoy effectively less freedom than good folks per the above reasons, and then take the hint and run 'good campaigns' with more meaningful character choices and letting the PCs be proactive and diverse in their belief. Success is measured by whether you and they learn that from the game.
3. I want to explore the darker side of human nature! This is why I run evil games. I actually have stronger beliefs on goodness than most folks in real life. I like confronting people with the consequences and ramifications of their actions in games to make them think. Is trading off part of your soul or good name or humanity worth it for that goal? How about long after you've achieved the goal? Success here is fuzzier, since games that actually uptake more roleplaying have less clearcut "win conditions" in general. But it's successful if it's enjoyable and if it causes people to grapple with moral questions.

The essence of sandbox play is following where the players lead, and it sounds like you're already doing that.

What adding randomness does is make the world feel more alive and larger than the thread that the PCs are pursuing/creating, allowing the players to make informed decisions about where they want to drive the game. You don't need to be constantly interrupting them with random encounters, but using them as a "spice" enhances the game and increases the players' knowledge of their environment, which itself increases the players' agency.

And don't worry about diversions derailing the campaign or slowing its advancement—the point of a sandbox is that there is no right path. If your players choose to diverge from their goals… it's because they have more goals than the one. If they want to advance one goal (not "the campaign"), then they will prioritise that goal even when distractions occur.

As long as your randomly-added distractions aren't mini side-railroads that force the players onto them, you can trust that they won't interfere with the players' freedom to choose their direction and speed. If the players choose to be distracted: that is where the campaign is. If they choose to not be distracted: that is where the campaign is.

Just remember that the campaign is where the players lead, no matter whether it was their idea or yours, and you can cut through this Gordian knot of worry about whether you "should" distract them with random things and whether the campaign is advancing at the "right" speed. There is no right speed or direction in a sandbox, only choices and outcomes.