Spiders are ambush predators, and would use the 5e hiding rules to do this
To go over your example:
A goblin hides in a bush, and a player must make a passive perception check to notice them, when they are within the vision of the player. Or they can make an active perception check to search for them.
Sort of. Players don't make passive perception checks, their passive perception is always on (it's a passive ability, not an active ability). If the DC to notice a hidden creature is a 12, and one of your PCs has a 13 passive perception, they will detect the hidden creature whether or not they're looking for it.
In my adventure, I had a spider hiding in the fog. The fog was so thick that vision was only 5 ft. If a spider is 20ft away presumably you cannot see it, either passively or by making a perception check, as it's no different to it being behind a concrete wall in terms of sight.
Keep in mind that perception is not only visual. While the players may not be able to see into the fog, they can hear the spider clicking around or smell it (I don't know if spiders smell bad or not). So while you are correct that they won't be able to see it, they may well be able to perceive it.
RAW, here is how you deal with the heavy fog:
A heavily obscured area—such as darkness, opaque
fog, or dense foliage—blocks vision entirely. A creature
in a heavily obscured area effectively suffers from the
blinded condition (see appendix A). A heavily obscured area doesn’t blind you, but you are effectively blinded when you try to see something obscured by it
This takes us to the 'Blinded' condition:
A blinded creature can’t see and automatically fails
any ability check that requires sight. Attack rolls against the creature have advantage, and the creature’s attack rolls have disadvantage.
So if the spider is 100% silent and has no smell, then you could reasonably assume that a sight check is required to perceive it. In that case, yes, it would be impossible for your PCs to detect it. But, very few things in this world are completely 100% silent, so RAW there is probably a change (albeit a very small one) that the PCs could detect it with an extremely good active perception roll (assuming they had reason to make one, but thick fog in a cave sounds like a good reason to me).
So how do you handle stealth in this instance? Because if a player moves within 5ft of the spider, it will see it automatically as it won't be behind cover anymore.
The details are up to you, but you can either determine that the spider is unhearable and unperceivable and as such has 100% perfect stealth, or that it may be perceivable and is potentially vulnerable to a great perception check.
To answer your specific question about how the combat would likely play out and what the spider itself would do
So can the spider actually hide? If it did hide using its stealth ability, how would you explain that to the players? The players won't be able to swing at where they think the spider is, because the players range is 5ft and they can see 5ft, so there would be no guesswork involved.
Looking at the relevant rules for hiding:
You can’t hide from a creature that can see you, and if you
make noise (such as shouting a warning or knocking over a
vase), you give away your position. An invisible creature can’t
be seen, so it can always try to hide.
So, your spider seems to fit the bill as a hide-able creature in a hide-able situation, so your spider can take the hide action, make his or her stealth check, and be hidden from the PCs until they detect it.
Obviously I want the spider to use the fog to its advantage, as it has 10ft of blindsight so can see 5ft further than the players.
RAW, that makes perfect sense. Just keep in mind that if your PCs are being quiet and are more than 10 feet from the spider, then the spider needs to make an effort to detect them as well. Basically, if both groups know there is danger in the fog, then they'll both be likely trying to hide from each other until first contact is made.
I considered it hiding to mean that it is carefully tiptoeing around the players, staying out of view but watching them so if a player does move within 5ft in technical game terms, it still can't see the spider because although the spider technically occupies the adjacent square, it is hiding somewhere else.
Hiding has a very specific meaning in 5e. When you're hidden, it means that you have a standing stealth score than needs to be beaten by either a passive or active perception check. As long as it is not beaten, and you don't make an attack, you are hidden. Once it is beaten or you attack, you are no longer hidden. An ambush predator like a spider would likely use this to get into position around the PCs (using its blindsight to stay 10 feet away but still detect the players), and then strike when one gets separated. Once it strikes, it has attacked and loses its 'hidden' status. After the attack, it can attempt to move away from all the PCs (to once again gain the heavily obscured condition so that it can hide at will). Then, it will take its next action to hide and move away from the PCs. It will roll another stealth check and it will use that to contest the PCs active and passive perception.
Keep in mind that as long as the PCs are in the heavy fog, they will be unable to see the spider until it gets next to them. Thus, they will have disadvantage to all attacks against it, and it will have advantage on all attacks against them. Also keep in mind that fog is generally dispersed with a wind of moderate or stronger intensity, so if one of your PCs has a wind spell they should be able to disperse the fog.
The fog cloud spell says (emphasis mine):
You create a 20-foot-radius sphere of fog centered on a point within range. The sphere spreads around corners, and its area is heavily obscured. [...]
Unlike nonmagical darkness, other heavily obscured areas created by spells such as fog cloud do generally prevent creatures from being able to see through them (unless there is some feature, spell, magic item, etc. that allows them to do otherwise - such as the warlock's Devil's Sight eldritch invocation when it comes to the darkness spell).
The distinction is that nonmagical darkness doesn't involve stuff blocking your vision, just the absence of light in that area. If there's a light shining down at point A and another at point B, and there's a bit of mundane darkness in between (but nothing else obstructing your view), then nothing prevents you from seeing the light at point B if you're standing at point A. In contrast, if there's a bunch of fog (whether mundane or magical) between A and B, it may be harder (or even impossible) for you to see B from A.
(Note that this relies on a common-sense interpretation of nonmagical darkness - the rules don't draw such a distinction, presumably because they expect people to apply logic and common sense in understanding how regular darkness works.)
As such, a creature in the middle of the fog cloud would be unable to see a creature outside the cloud, just as the creature outside would be unable to see the creature within the cloud. If one attacked the other, the attacker's advantage from being unseen would be canceled out by their disadvantage from being unable to see their target.
Best Answer
So I have not used Roll20 in a couple of years (or play D&D over those years unfortunately), but here is what I used to do for my players in situations where they were in the dark without a light (or otherwise had very low visibility). I would turn on Dynamic Lighting but turn off Global Illumination. Then for each of their tokens I would go to the advanced setting, make them emit light out to 5 ft and unselect All Players See Light. That way each player could see what was right next to them but would not see the other players if they were far apart. I also asked my players to not move too quickly (especially outside of combat when they are not taking turns) and they were pretty good at stopping or asking questions when they butted up against something or someone. Also you might try to incentives them asking questions in ways that do not reveal information to other players. "What am I seeing next to me?" instead of "IS THIS HIDEOUS CREATURE HOSTILE!?". In 5E they have ways to reward players for role playing well which could be adapted for this.
The disadvantage of this approach in your case is that if you only want small sections of the map not visible you will have to create a grid of light sources across the map and then turn off the ones where the Obscuring Mist should be.
That is what I have done in the past. Looking at the current offerings I think Advanced Fog of War is what you want. First of all, you can set the sight for each token (under the token's advanced settings):
And they can only see what is in view for their token:
My suggestion would be to turn on advanced fog of war, set each token's sight to 5ft or however far you think they should be able to discern, use the Reveal Areas tool to reveal the whole map, and then use Hide Areas when something like Obscuring Mist is cast:
Lastly, if the players need more than what they can see from the map when they uncover an area, I would "whisper" to that player. To help me be able to whisper fast enough I type up (in a text editor or word processor) whispers for things I know I want to share at some point. Then I can cut and paste them to the appropriate players. Of course the players are always going to ask about something that I have not prepared before hand, but hopefully these are not too frequent and all the players are patient while I type the answers to these questions.