Switch – Lutron Fan & Light dimmer switch wiring

No way from here to know which wire fed the light or fan, but you have a continuous hot feed into the box that fed the receptacle, light and fan. The third wire, maybe it fed the fan or maybe something something else farther down the line, if the latter that needs to be tied in 100 percent to the hot feed. All whites still tied together, blacks to the dimmer from the light and hot. All grounds tied together set with a screw to the metal box. Securing the ground to the box is not needed if the box is plastic.

Before I got too wordy, here is a sketch

Still have to learn to format better

The sketch is a little rough, it should get the idea across I hope

You said

At the switch box there is a light switch that is for the vanity light and one that controlled the old fan.

So you are removing this switch and replacing it with the new one? It isn't exactly clear what you are trying to do. A drawing or pictures would help.

If the new fan has a light in it then you need to use the either the black wire for the fan or the light and the red for the other. The new switch should be fed with a "hot" wire (one that is energized all the time) and one of blacks goes to the light and the other to the fan.

Hope that helps.