Electrical – New Light Fixture with Old Wiring

You probably should have paid more attention to what was connected to what. In all likelihood, the "sw" bundle has a wire that was carrying hot (power) from the unswitched wire at the fixture to the light switch, and the other wire was returning that power from the switch. The switched wire should be but often is not re-colored red in that application (ie, a back/white/ground cable running to a switch should have red tape or paint on the white wire ends if it's a "switched hot" rather than a neutral.)

That does not meet current code (which requires a neutral wire at the switch) but met older codes and was very common (a switch loop.)

As such, the expected configuration that will probably work if things are wired normally would be cable blacks together, white from unswitched cable to white on fixture, white (recolored red for clarity) from switched cable to black on fixture, and grounds all together.

You said

At the switch box there is a light switch that is for the vanity light and one that controlled the old fan.

So you are removing this switch and replacing it with the new one? It isn't exactly clear what you are trying to do. A drawing or pictures would help.

If the new fan has a light in it then you need to use the either the black wire for the fan or the light and the red for the other. The new switch should be fed with a "hot" wire (one that is energized all the time) and one of blacks goes to the light and the other to the fan.

Hope that helps.