This is a bad idea.
The maximum power rating for low voltage lights is lower than that for normal lights because no transformer is 100% efficient, so some allowance is made for that inefficiency in the specifications. In your case, those four lights draw 200 W plus whatever overhead the transformers add, so you're drawing more current than the switch is designed to handle. More current through the switch means more waste heat generated by the electronics in the switch, which leads to higher temperatures inside the electrical box, which in turn will shorten the life of the switch1.
To answer your question about the brightness: I'd expect that they'd get less current than they should normally which would make their light both dimmer and redder. Generally, you have bright lighting in bathrooms, so even if it weren't a problem for the switch, it might not be something you'd want.
1. If you look at the specifications for the switch, you may see de-rating information for using two or more of them in a single electrical box; this de-rating is caused by the waste heat generated by the switch.
It sounds like you have a fixture controlled by a conventional dimmer and are using CFL non-dimmable bulbs.
Standard dimmers were designed for incandescent bulbs. Many CFL and LED bulbs are non-dimmable. Some are dimmable, but only work with special dimmers designed for their electronic circuitry (conventional bulbs have no circuitry, only simple filaments). Some CFL or LED (not many) will work with a standard dimmer.
First you need to make sure the bulbs are dimmable, and then that you have a compatible dimmer.
In general, CFL and LED bulb packages indicate whether they are dimmable or not. If it does not say dimmable, it is probably not. Dimmers also indicate whether they are intended for CFL and/or LED bulbs. Again, if it doesn't mention CFL or LED, it is probably not compatible, unless you have bulbs that specifically say works with standard (or conventional) dimmers.
Check the fixture by using an incandescent bulb. If it works, either go with incandescent, find CFL or LED bulbs that work with standard dimmers or swap out the dimmer.
If the incandescent doesn't work, you have a wiring problem or a defective fixture.
Best Answer
Because you have too much lighting load for the power supply (not transformer*) to cope with, so it is crowbar-ing.
Reduce load. This is a good time to think LED.
* "Transformer" is a proper term for a very specific thing, which this is not. Transformers are quite heavy and withstand short-term overload very well. This weighs 6.4 oz. for 150W, and fails instantly at 100W. They can call it an A-10 Warthog if they please, but it's actually a switching power supply. As such it has a protective circuit (crowbar) which is tripping for whatever reason (defective? Misapplied? Loads you don't know about?)
The RU mark is a good sign, but still it's only rated as an electronic component, it would need a UL mark to be legal for direct use in mains wiring.