Electrical – install new light on existing switch

The first rule of novices replacing fixtures (or anything) is only mess with the wires that go to the old fixture. Don't touch things that are outside project scope, no matter how interesting they seem. That is to say, resist the urge to learn electrical by dismantling your house. Get a DIY book on the subject and read it through.

Safety Ground wires don't need much thinking, because they all go together.

The clump of all-white wires is certainly neutral, and an appropriate place to terminate the white wire(s) from your fixture. What is always true is that the place the old lamp's white wire went, is correct -- barring previous incompetence of course (and novices are in no position to judge competence).

What remains is the switched-hot wire(s) from the fixture, presuming you want the lamp to be switchable. If that wire comes from the ceiling box, it is usually a solitary (connected to nothing else, who else would want a wire that is only energized with the lamp on?) This will be the wire that went to the lamp's black wire before.

Color coding is often not that helpful. Switched-hot is a type of hot, and all hot wires can be any color not reserved for neutral or ground. Installers get no choice of color since all cables come only black/white or black/white/red. However by convention it's preferred to use red for that when possible. So red is a good bet, and since you say the old lamp's wires went there, that all jibes.

Black and white

Your idea of connecting both lamp whites to the white cluster, and one black to supply black and one black to supply red -- the result was one lamp is always-on, and the other is switched. You may have missed that if you weren't paying close attention. (Light works on a log scale like sound, so half the light is only 3db dimmer. 3db isn't a lot).

In this LED age, it costs so little to run small lights that I have started wiring some lamps to be always-on for safety and put small 3-4 watt LEDs there. ($3-4 a year). In a 2-lamp fixture I'd make one lamp switched and put a very strong LED in it, so you have low/high instead of on/off.

If it also has switched receptacles

Lastly, if you have split bulbs black and red, and it still doesn't switch, there's a fair chance the house also has switched receptacles... and a novice (no names mentioned) replaced the receptacles and wasn't paying attention to the breakaway "tabs" between the two sockets on the hot (brass screw) side. That would defeat the switch, causing the red wire to be always-on.

The cure is to search all the receptacles in the room, looking for red wires or anything abnormal. In such a location (there may be several), get some spare receptacles (they are $3) and break the brass-screw-side tab off. If doing this causes downline receptacles to stop working on both sockets, it was the wrong thing to do; tabs can't be de-broken so swap in a spare receptacle. Needles to say, cut power at the breaker and confirm it's off before opening up a receptacle.

Assuming this is the setup you have described:

You will need to pull a new 12/3 cable to your new 3-way switch. You will also have to replace your existing switch with a 3-way switch. Your wiring setup will look like this:

Because your lighting circuit's power is supplied by a conductor in a 12/3 cable, the figures above were modified as follows:

POWER SOURCE / 2-Wire Romex with Ground (i.e. 12-2) - I modified these to show your 12/3 wire with your RED supplying the power to your lighting circuit (since you described the BLACK in your 12/3 as being spliced into a bunch of other black conductors). From here, everything else should be one-to-one.