NO PUTTY!
You have a loose electrical connection and a loose mechanical connection.
Mechanical
First you need to fix the mechanical attachment mechanism. It may be a loose nut on a threaded pipe holding the individual fixture to the base. It could be some other locking mechanism, such as bolts or screws. In some fixtures, the sockets are held in by pressure fits, which are harder to repair.
In either case, you probably need to take at least the lighting unit off of the fan, and maybe the whole fan down, depending on how it is put together. Before you do anything, turn off the circuit to the fan, not just the wall switch or the remote control switch. Once the wiring is exposed, double check that all wires are not hot by using a non-contact tester.
You need to restore the original attachment method, not just a glue or putty to wedge it in place. Fan vibration will almost certainly cause a makeshift repair to break again.
Electrical
It sounds as if you have a loose wire. You need to make sure that all wires, hot neutral, and ground are all connected (in the US, these are black or red, white, and green or bare). The ground may be established by physical contacts among the metal parts, and restoring the mechanical connection may fix an intermittent ground. Sometimes a wire may be under a wire nut, but not well connected. If there is no obvious disconnected wire, remove and check wire connections under wire nuts.
Replacement
If you cannot make the physical socket stable as originally designed, you need to replace the fixture (or at least that part of it). Underwriters approval (a safety standard) is dependent on the structure as designed. If you cannot restore the mechanical structure, you probably need to replace the entire fan.
Looks like it might be an EJ500 Astro In-Wall Timer, or similar device. Here's a link to the Installation Instructions (PDF).
According to the instructions, the switch at the top of the device is an "air gap" switch. Which is "designed to turn power off for routine maintenance".
You remove the battery holder "by prying left and right of the holder
with the tip of a flat screwdriver, then pry the battery holder up."
There's a another document available, describing the battery installation/replacement (PDF).
I'm not familiar with this device myself, so I'm not sure if it's fixable or if it should simply be replaced. You might try changing the batteries and see if that helps, as the instructions mention changing the batteries if the display is dim.
- Replace the batteries if the display is dim or displays “lobAT”
Best Answer
It turns out that the previous owner had 3 100 watt bulbs in a fixture only rated for 60 watt bulbs. This caused the paint on the metal to peel/adhere to the glass. It eventually fell off on its own since I left the bottom screw off.
One of the sockets was blackened from the heat, so all 3 of these things in my house are getting replaced ASAP.