1. With HPS, no for smaller bulbs on bigger ballasts.
Discharge lights are current devices. They are non-linear (behave almost like a dead short) and so, must be externally current-limited. That's the whole point of a ballast or driver.
HPS bulbs work at a specific current in one of four working voltage ranges (page 9): 52V, 55V, 100V or 250V plus or minus 15%. To swap, the bulbs would need to be in the same voltage range. Too small a bulb, the bigger ballast would force the higher current through the bulb, burning it out. The reverse, the bulb might work, but might not warm up properly.
By comparison, if you stick a VHO 8' fluorescent tube in an HO fixture, you get HO light output from the VHO bulb.
2. Nope! Different and incompatible, especially ignitors
You absolutely cannot interchange various types of discharge lighting - not least because the way the ignitors work is very different.
Here, HPS is an odd duck. It has an ignitor which fires a 2000-3000 volt spike to ignite the bulb. The supply is constant-current, so it will increase voltage under no load. This triggers a VBO, which flows a burst of current into a winding on the ballast transformer, inducing a spike to the bulb.
They make special MH bulbs made to run on HPS fixtures; these are resistant to the spike and are tuned for HPS voltage and current. They cost 50-100% more than a stock HPS or MH bulb.
Other than that, facility managers don't intentionally use the wrong bulb. Nobody will support that, so if you want to try it, you're on your own. It will either not work, give poor life, burn out the ballast, or blow bulbs - and these suckers run very hot and you don't want molten sodium raining on your head. Some are even reputed for starting fires.
3. LED replacements must be designed for that.
You linked a compromise screw-in LED meant to go into an HPS or other HID light without removing the ballast. If they are designed, (UL) listed, and labeled for that use, it's safe and legal. Otherwise, it is not. The operative word is "compromise".
HPS is more lumen-efficient than any but the most efficient LEDs on the market - if you like yellow light. And if you're trying to light the inside of a sphere, which you might if you're growing pot. Reflectors suck. If you really want a cone of light (and most people do), LEDs win big, because they inherently emit a cone of light. They are far more lumen-efficient inside that cone, and weak reflectors don't help enough.
Of course, if arranged like a "corn cob", the advantage is utterly lost. SMH... Those are for people in a hurry looking for a screw-in replacement that uses their same reflectors and lensing. But wait. Why keep the old fixtures? Do you really want to keep the old ballast? No, it wastes energy and is a maintenance item. You want to direct-wire existing line voltage to the LED device. Do you honestly foresee rolling back to actual HID lighting? Never gonna happen. Throw the fixture in the attic, get a modern LED fixture for 1/3 the cost - and don't look back.
Standard practice is replacing a 400W HID with a 6-tube F32T8 fixture. 4' LED replacement tubes are a commodity item, and prices are in free-fall -- $8 last I checked for a top brand. $48 and you have a HID light's worth of LED light. Hit up Craigslist for a used fluorescent fixture, yank the ballast and rewire direct - you're in business. Given how cheap LEDs have gotten I wonder why anyone would fool around with HID lighting anymore.
Real fluorescents are even cheaper if you get eBay ballasts, and the superb tubes now available.
This is extremely common behavior for old fluorescent ballasts, though the reason for it depends on exactly what type of ballast it is.
If it is a really old model that uses a replaceable "starter" cartridge, it is likely that the starter needs to be replaced.
If your ballast does not require a starter, the issue most likely the lack of a ground plate behind the bulbs. "Rapid Start" magnetic ballasts require the metal plate that makes up the back of the fixture to be grounded, which provides a pathway for the "radiated" high voltage current that starts the lamp.
If the fixture is already properly grounded and this is still an issue, then it is likely that the ballast is failing and is just too weak to provide a good starting current.
Best Answer
Ballast replacement is fairly straightforward. I still prefer fluorescent tubes because I can easily find quality fluorescent tubes in 90+ CRI (in fact Menards stocks them), and I have serious doubts about the real quality and longevity of the LED "cheapie" tubes.
Start with a "ballast disconnect"
For this you need to shut off the circuit breaker supplying power. You then put the ballast disconnect at the point where the power supply (typically 12-14 AWG wire) meets the ballast (18 AWG wire). Some disconnects care which side is the heavier wire, i.e. the yellow Wago types. If there are 2 power cables going in and out, then either use the Wagos (which provide for that) or pigtail it.
Once that's done, you'll be able to service the fixture without turning the breaker off. Just unplug.
Next select the bulbs
Those look like 48" bi-pin tubes (the most common one). From looking at the "old" Keystone ballast, your old ballast was for F32T8 tubes, 1" diameter (T8). Which means it's not that old. Don't use F40T12 tubes, 1-1/2" diameter, they will fit but they are not electrically compatible unless you change to a T12 ballast.
Look at the existing ballast
The big label on the old Keystone ballast says your bulbs are instant-start, with 1 wire per end of the lamp. Gotta watch out for that 2-yellow 2-red 2-blue color scheme as it's also used by 2-lamp rapid-start ballasts. This is an example where the diagram really matters.
Check the new ballast also, the wiring may be subtly different. If the Lowes clerk picked your matching ballast by the same wiring diagram, that's handy, but double-check the type of tube it's made for. Here are generic drawings of ballast wiring; yours may vary. For instance yours wires like two separate 2-tube instant-start ballasts.
Historically 4-lamp fixtures often had 2 ballasts. There is a mounting site in your fixture for a second ballast. Aside from using the more common 2-lamp ballast. it also let you feed them from 2 switches for "high" and "low".
If the ballast is a goner, you'll have no trouble finding 4-lamp ballasts in either instant (1 wire) or rapid-start (2 wires). Rapid-start gives better tube life; a sub-type is programmed-start which gives superb tube life - something to think about if the fixture is hard to access. But if you switch to rapid-start, you'll need to change the lampholders (tombstones) on the end of the tubes to a non-shorting type.
Edited significantly based on updated info