I have tried this in two ways in the past. I think of the two, only one will be of use for your objective.
PC Villain in the Group
Create the villain with its player and discuss what their villainous goal actually is. Ensure the player can and will commit to being a villain. Their goal should require the villain to need to be close to or involved with the players to achieve. You will probably want the nature of your villain to be a big revelation at some stage of the campaign, so they need strong hooks to tie them to the heroes, and to tie them to what the heroes are trying to do.
Of course, the heroes will want to achieve a positive outcome in the scenario, while their supposed "friend and fellow hero" is actually trying to ensure a negative outcome. This ensures challenge for each player without necessitating that the players be targetting each other, limiting the sense of mutual dependence, or requiring that the relationship be entirely based on lies.
In truth, the variance between the heroes and the villain might only a slight difference in ideology, the burden of a dark family secret, or the interplay of noble but misguided intentions.
Example:
*Over time the group learns that an ancient prophecy claims a soul-collecting terror from ages past will rise again to plague a small village on the ancestral lands of one of the characters.
As they uncover more and more about the prophecy, they realize that the time will soon arrive. They must act now. They become desperate to discover a way to prevent or guard against the soul-collector.
Throughout this period, the villain is aware that it was their many times great grandmother which unleashed this foul curse on the village in the first place to spare her own children from the ravages of the soul-collector. If the villain chooses to delve deeper into the lore of the family, they will discover much more about the monster than the heroes, including that it was bound by the family to both stop greater depredations and to extract special favors from it. They can, if they have the will, learn how to bind it, and become tempted by what the being will offer for even a little bit of freedom. They will also learn that if the monster is balked it will come for them, their siblings, and other members of that generation within the family and nothing will be able to stop it. Worse, if they reveal the family secret they will be disowned, the monster will take back all the gifts it provided the family, and it will be set loose to kill indiscriminately. To ice the cake, if the villain chooses to master the rites involved, they can gain direct control of the monster and earn their own, very special favors from it.
The heroes will of course be committed to stopping the creature, and their flawed understanding of the situation will enable the villain, if they stick close to the heroes, to monitor progress and to try to redirect it when necessary to prevent success.*
Requirements
This style of scenario best pits character goal versus character goal, but not character versus character explicitly. It does not need to require that the villain receive lots of useful extra knowledge, nor that they go off for extended periods to plot or take care of things off-screen. Moreover, it allows all the players to play toward a goal of uncertain resolution, rather than forcing one of them to be a glorified NPC just there to trick the players. A great scenario will have lots of conflicting emotions on both sides of the group as the course of events plays out.
Key Points
- Everyone plays in the scenario, no ringers or PCNPCs
- Each has a related goal that they may or may not achieve
- once play starts there is no meta-fiddling with the villain's
knowledge or activities
- the heroes may remain focused on ending the threat little knowing
they have a viper in their midst
- the villain has the burden of keeping their secret agenda of foiling
the goal of the heroes without getting caught
At the end, there is a chance for a dramatic conclusion when the duplicity of the villain ultimately comes to light. If the bonds between all the PCs are tight, the heroes may even be swayed to the villain's side...
The essence of sandbox play is following where the players lead, and it sounds like you're already doing that.
What adding randomness does is make the world feel more alive and larger than the thread that the PCs are pursuing/creating, allowing the players to make informed decisions about where they want to drive the game. You don't need to be constantly interrupting them with random encounters, but using them as a "spice" enhances the game and increases the players' knowledge of their environment, which itself increases the players' agency.
And don't worry about diversions derailing the campaign or slowing its advancement—the point of a sandbox is that there is no right path. If your players choose to diverge from their goals… it's because they have more goals than the one. If they want to advance one goal (not "the campaign"), then they will prioritise that goal even when distractions occur.
As long as your randomly-added distractions aren't mini side-railroads that force the players onto them, you can trust that they won't interfere with the players' freedom to choose their direction and speed. If the players choose to be distracted: that is where the campaign is. If they choose to not be distracted: that is where the campaign is.
Just remember that the campaign is where the players lead, no matter whether it was their idea or yours, and you can cut through this Gordian knot of worry about whether you "should" distract them with random things and whether the campaign is advancing at the "right" speed. There is no right speed or direction in a sandbox, only choices and outcomes.
Best Answer
This is a very fun idea. Here are some of my thoughts, which you hopefully find useful. Feel free to use one or more ideas, or none if none suit your campaign.
Your Co-GM's Character Has Other Things To Do
You may both know the major things that affect the world, but your characters have much more immediate things they have to take care of; and that neither of you have to share with the other.
Without knowing details about your homebrew world, here are some examples that illustrate what I mean:
If the next major event is an insurrection of the people against their tyrannous vampiric oppressors, one party might be a pocket resistance smuggling weapons and warfare paraphernalia; the other party might be a collection of people, meeting for the first time on the ferry away from the city, trying to escape on the eve of the rebellion.
If the next major event is the invasion of the villain and their army, one party might be in the thick of things, present at the moment the walls were breached and witness to the fall of their kingdom; the other party might be returning to the city after the scourge has passed, coming home to ash, dust, and rubble.
If the next major event is the return of a magical winter, one party might be busy paying absolutely no attention to it while fully knowing it's coming; the other party might be busy telling everyone they meet that it is coming.
The point is, just because you have a "shared destiny" in the world, doesn't mean you will experience it the same way, or experience the same things along the way.
Go for Simplicity
Don't decide over too many plot details with your co-GM. Go for simplicity.
Instead, set up something like "prompts" -- a small phrase that fully describes the entire core of the next "major event."
That isn't to say don't flesh out the world with him; you still do that. This doesn't mean don't agree on how the villain will ruin everything. It also doesn't mean don't spend hours designing the campaign with him.
The idea is only to set up a string of "prompts" to form the backbone of your metaplot, that roughly describe (1) what is happening, (2) where each party is, and (3) why they care about what is happening.
Design The "Major" Events Well
Imagine scenarios where, knowing what's coming ahead, when that moment arrives, each party experiences it in starkly contrasting ways. The goal is either to put in irony, serendipity, or revalation -- or something else that feels awesome.
On the King's coronation day, one party is standing beside the King; the other is about to shoot him in the head (and get away with it)
On the day of the villain's prison break, one party is summoned to give chase; the other, oblivious, is looking at a list of jobs at a tavern when a cloaked, hooded figure speeds past them on a horse, pursued by a party of mercenaries
One party gets a quest from an NPC; the other party comes across the same NPC later and is revealed to be a demon
But once you've decided roughly what's going to happen next, stop there. Put an emphasis on the fact that the next "major event" affects the two parties together very tangentially, but affects the environment a lot, so both sides have a lot of room to be affected by and interpret the circumstance independently of each other.
Forget The Metaplot
Create an independent, self-sufficient story arc independently from your overarching plot. Then, at times that your private story arc interacts with the metaplot (during these "major events"), flesh out how those two lines clash and merge (but only by yourself, your co-GM doesn't need to know).
This is a good way to catch your co-GM's engagement in your story, and keep them on their toes. Because you're GMing as if there was no metaplot, they will have no way to use their knowledge to benefit themselves, even accidentally, unless they try really, really hard to break the campaign.
The Party Will Surprise You Both
It goes without saying, but you may not have to keep your co-DM on his toes yourself, because you yourself will be constantly adjusting to your players.