I don't play a lot of 3v3 and 4v4, but I would imagine the later tech transition is in part due to the much larger army you can bring to bare in early stages (having 4 people to draw on). I'm not sure how to help with that, but I can give you the 1v1:
Tech transitions should be part of your overall strategy. If your strategy is: Speedling into Infestory/Baneling into Ultralisk (the TLO ZvT), then you have clearly defined goals as to when to tech. You know that you'll have early harassment with speedlings, you switch to Baneling/Infestor as he starts to get Medivacs for his MMM, you'll upgrade melee damage, and once he gets his factory up and producing you'll move to Ultralisks.
Day9 once said "every [good] strategy has a beginning, a middle and an end." This was true even at the beginning of Beta when Zerg players went just Roaches. The strategy then was: I'm going to open Roaches. For my midgame I'm going to get Upgrades, Roach Speed, Burrow and Tunneling Claws. For my late game I'm going to get Roach HP Regen while unburrowed (since removed). Its not hard to see why people thought Roaches were overpowered at the time, but the key thing here is that even in this simple case he has ideas of when he wants to tech; its part of his core strategy.
This is not to say you won't have to adapt to what your opponent is doing. If your strategy is the TLO ZvT hellion harassment, maybe you build some Roaches before teching to Infestor. However, the key to execution is knowing that even before you've completely shut down that harassment you need to be upgrading your lair and getting Infestors. Just because you adapt doesn't mean you have to compromise your strategy.
I'm going to give you one last example: Terran MMM, because its actually well defined:
- I'm going to wall off and open 3 Rax
- Once I have a nice clump of units I'm going to expand
- My mid game is going to be getting Stim and Medivacs
- Once I have a sizable force I'm going to start breaking map control
- For my end game I'm going to add on factories and start pushing siege tanks.
The reason MMM is such a popular build is that is very easy to do and it has a very nice mid and late game transition. Once you have your 3 Barracks you think about Expo. Once you have Expo you go to Medivac for your Stim. Once you've moved to late game you're backing up with Tanks. At all points you are strong, and at no point do you stop using the unit producers you currently have.
A long time ago I wrote about scouting, and I really think it'd help you to go back and take another look.
The purpose of scouting is not omniscience. Getting that level of scouting would cost too much time and effort and just wouldn't pay off. Instead, scouting is largely to get an idea of what they're doing at the current moment.
You don't need to know every time they build a Starport, or take a new Expansion. You don't have to be on the ball with your army half way there to counter whatever crazy tech switch they're about to pull, if that were the case this game would be all about deception and hiding your next building.
Keep this in mind when you're considering the following.
The number one way people scout is skirmishes. This may seem like a strange concept as it makes sense to want to know what your opponent has before engaging them, but usually the way player figure it out is those quick little back-and-forths in the middle of the map, or the drop in the back of the base, or a fly by with a group of Mutalisks.
The reason these are so telling is they give you an idea of what he has right now, and where he's going. Notice he added on a couple Vikings since your last fly by? Maybe you see a Colossus laser eat your dropping Marines. Personally, I always find Thor missiles to be a good indicator that he has Thors... and probably more coming.
With the exception of maybe some Dark Templar rush or Hidden Void Rays there isn't a lot your opponent can build up in secret that will make a huge difference; and for those sometimes cheesy plays, there is no cheese that can't be handled by standard play. Unless you're going for some crazy Cheese yourself you should be in a good place.
So my advice to you: Stop worrying so much a live a little. Sure maybe when you suicide a Marine for intel he doesn't see a Carrier, but you know what? Unless you're doing something crazy, that's ok. You'll catch them before he has more that 2-3.
Alright Tzenes, if trick plays aren't some unmanagable thing, why am I Scouting?
Scouting is supposed to give you a feel for what your opponent has. Its supposed to give you answer like: BioMech on 3 Bases. Its not for things like: OMG he built a tech lab on his second Starport! Starcraft isn't precise enough a game yet that you can know exactly what that means and how to counter it. Maybe for the first 3-4 minutes of the game, but after that point you just need a general feel.
No Zerg player can go Muta/Infestor/Broodlord/Ultralisk in a undecided game, so chances are if he's producing one of these you probably know what to expect and where he can go next.
Best Answer
Don't ask yourself "how" to defend but ask yourself "why" that he plans a medivac drop.
Impressive medivac play is shown in Day[9] Daily #248 - Dealing with Duckload (14:17 - 27:00),
let's see at what we have to note and can learn from watching this video:
Ugh... What went wrong at Protoss' side?
Ah... What could Protoss have done instead?
As for how, you'll find the answer in the Help in-game or at Liquipedia II, there isn't much more to say:
If you learned why you'll have a better composition to handle the issue.
We could list any combination, but inspecting what went wrong will learn you how to do that yourself...
Scouting and map control are key. Don't mindlessly attack, but know what your opponent is doing...
Note: If you plan to use Tanks, have something to counter air or else he'll simple drop on your tank.