Diablo – penalty for switching followers

No, no penalty involved. I think the reason for the cast time is simply that you can not do this easily during busy fights. Other than that, though, you should be fine.

Diablo 3 has no concept of "In Combat". (This is the main reason your "Massacre" bonus can end while you're still fighting tooth and nail)

Everything is calculated when it happens. If you've got the horsepower to swap gear between each kill, all the more power to you, but it's a lot of work for a lot of very minor gain.

Unlike switching skills, there appears to be no penalty to switching equipment.

In the discussion leading to patch 1.0.4, Blizzard asked for solutions to Magic Find gear swapping. Some of their own suggestions included penalties for said gear swapping. In the end, they did not change the way gear swapping works. They simply added a 300% MF/GF cap.