It's never the best of an idea taking the cake out of the oven during its cooking as you'll probably knock it a bit and knock the air out and then it'll cool down a bit and so take longer too cook. Saying that, the whole thing is when you take the cake out. As a rule don't take your cake out until at least half-way through the cooking time, that way the starch will have started to set. If you take your cake out multiple times 5 minutes before, say, it don't make to much of a difference as by then the cake is practically cooked.
Some cakes are more susceptible: foam cakes (genoise, chiffon, etc) are the most likely to feel the full force of you taking the cake out of the oven as it will likely deflate all of the air you've worked so hard to create, however, butter cakes (pound, fruit, etc) wil not be so effected as these are a lot denser and rely more on chemical leaveners.
There are lots of ways to check if your cakes done. If its a cake with quite a thin batter, open the oven door and give it a tiny wobble (not a great big shake to deflate it) and if it wobbles its definitely not done, if it doesn't wobble you can progress to the next step which is poking in a skewer or knife and if it comes out clean it's done, use this technique for all cake types. Another way is when the cake looks golden (unless it's a chocolate cake), press your finger down on the cake and if it springs back it's done. A lot is made about not opening your oven door, in modern ovens today, this doesn't make much of a difference as they can quickly get back up to temperature, particularly if you just open the door a crack. On a final point having a reliable oven temperature is one of the most important steps. Buying a little oven thermometer is a good investment, they're very cheap and will save a lot of hastle
Because of the dairy products, you shouldn't leave it un-chilled for more than 4 hours.
It's always best to cover your food to prevent dust (or cat fur, for example) from covering it.
Unless the frosting is really wet and gooey (which might run into the cake), it's OK to frost now.
Best Answer
Most cake recipes I have seen give an approximate baking time and also instructions for determining doneness by observation (the "toothpick comes out clean" sort of guideline). When the criteria for doneness are met, it is assumed you will take the cake out of the oven immediately.
Very often there are further instructions about cooling, removing from the cake pans, frosting, etc. But they don't expect you to leave the cake in the oven (whether turned "off" or "on") past when it is done. Leaving it in the oven while the oven is cooling down would just introduce too much variability as ovens would cool at different rates.
I agree with your speculation that cake problems must be from some other cause.