Lighting – use a 3-way light bulb in the 60 watt lamp

This is a bad idea.

The maximum power rating for low voltage lights is lower than that for normal lights because no transformer is 100% efficient, so some allowance is made for that inefficiency in the specifications. In your case, those four lights draw 200 W plus whatever overhead the transformers add, so you're drawing more current than the switch is designed to handle. More current through the switch means more waste heat generated by the electronics in the switch, which leads to higher temperatures inside the electrical box, which in turn will shorten the life of the switch¹.

To answer your question about the brightness: I'd expect that they'd get less current than they should normally which would make their light both dimmer and redder. Generally, you have bright lighting in bathrooms, so even if it weren't a problem for the switch, it might not be something you'd want.

^{1. If you look at the specifications for the switch, you may see de-rating information for using two or more of them in a single electrical box; this de-rating is caused by the waste heat generated by the switch.}

I'm assuming this is a 100 watt equivalent bulb, that is actually only running 18-20 watts. Using a 100 watt equivalent will be safe as they run much cooler than even an incandescent.