How to clean up white dust and spots “blooming” on rubber

I don't usually bother letting it warm up. I just use a mixture of a mild kitchen cleaner and warm water on a paper towel. If you're careful, and don't let any moisture pool up, it seems to evaporate before it freezes to the inside.

If you're just trying to combat a smell, try putting some baking soda in there.

As for how often to clean, I usually do a purge of the contents of the freeze 1-2 times a year, and do the cleaning then.

You must live in a warm humid climate. If so, then these are indeed fungi or mold (same thing). Fungi reproduce via spores. The spores are in the air all the time, so you cannot get rid of it, you can only attempt to manage it. They grow on certain surfaces in warm, humid conditions. They are growing even when you are home, but your activities disrupt the colonies so they remain small and invisible. Left undisturbed, the colonies become large enough to be visible as white spots.

To manage fungus growth, encourage environments they dislike-- cold and dry. Very difficult in warm humid climates unless you can run air con all the time. At least avoid introducing any additional moisture into your environment. Avoid using surfaces in your residence that they like, such as cloth, in favor of surfaces they dislike, such as hard materials. Of course, it's difficult to wear hard materials.

Even on surfaces they do not grow, their spores sit and remain dormant until a disturbance moves them to a favorable surface. Regular cleaning of all surfaces with a mild chlorine bleach solution will help keep spores and colonies in check. For surfaces and clothing where bleach would be damaging, wash with the hottest water possible. For materials where hot water would be damaging, use anti-microbial soap. Use this soap only as needed, as regular use promotes the development of "super bugs" that are resistant to normal controls such as methicillin.