The first rule of novices replacing fixtures (or anything) is only mess with the wires that go to the old fixture. Don't touch things that are outside project scope, no matter how interesting they seem. That is to say, resist the urge to learn electrical by dismantling your house. Get a DIY book on the subject and read it through.
Safety Ground wires don't need much thinking, because they all go together.
The clump of all-white wires is certainly neutral, and an appropriate place to terminate the white wire(s) from your fixture. What is always true is that the place the old lamp's white wire went, is correct -- barring previous incompetence of course (and novices are in no position to judge competence).
What remains is the switched-hot wire(s) from the fixture, presuming you want the lamp to be switchable. If that wire comes from the ceiling box, it is usually a solitary (connected to nothing else, who else would want a wire that is only energized with the lamp on?) This will be the wire that went to the lamp's black wire before.
Color coding is often not that helpful. Switched-hot is a type of hot, and all hot wires can be any color not reserved for neutral or ground. Installers get no choice of color since all cables come only black/white or black/white/red. However by convention it's preferred to use red for that when possible. So red is a good bet, and since you say the old lamp's wires went there, that all jibes.
Black and white
Your idea of connecting both lamp whites to the white cluster, and one black to supply black and one black to supply red -- the result was one lamp is always-on, and the other is switched. You may have missed that if you weren't paying close attention. (Light works on a log scale like sound, so half the light is only 3db dimmer. 3db isn't a lot).
In this LED age, it costs so little to run small lights that I have started wiring some lamps to be always-on for safety and put small 3-4 watt LEDs there. ($3-4 a year). In a 2-lamp fixture I'd make one lamp switched and put a very strong LED in it, so you have low/high instead of on/off.
If it also has switched receptacles
Lastly, if you have split bulbs black and red, and it still doesn't switch, there's a fair chance the house also has switched receptacles... and a novice (no names mentioned) replaced the receptacles and wasn't paying attention to the breakaway "tabs" between the two sockets on the hot (brass screw) side. That would defeat the switch, causing the red wire to be always-on.
The cure is to search all the receptacles in the room, looking for red wires or anything abnormal. In such a location (there may be several), get some spare receptacles (they are $3) and break the brass-screw-side tab off. If doing this causes downline receptacles to stop working on both sockets, it was the wrong thing to do; tabs can't be de-broken so swap in a spare receptacle. Needles to say, cut power at the breaker and confirm it's off before opening up a receptacle.
Best Answer
This is all LED related.
Option 1: You are changing from incandescent to LEDs.
The new LEDs you are using do not play well with the dimmer, smart switch or other smart electronics currently installed. Those old electronics depend on a unique characteristic of incandescent bulbs, which most LEDs do not bother emulating. When you change the last incandescent to LED, that breaks the smart device. The fix here is to address the incompatibility (change one or the other) or add a Lutron LUT-MLC, which provides the characteristic the old dimmer/etc. is looking for.
Option 2: You are using incandescents, not LEDs.
Most fans have a safety circuit designed to prevent you from overlamping the light sockets, meaning put in bulbs that are too large and make too much heat. These would damage the fixture or start a fire.
Adding the fourth bulb pushes the power draw over the limit, resulting in the safety circuit doing its job.
The cure here is to use LED lights, which draw vastly less power, and can give plenty of light for far less heat. The fixture cares about the heat, not the light.