Plumbing – Curb water shut off access pipe too high

How can I get it to close fully?

The plastic handle is expendable. It will break long before the valve spindle. This is good. I would use an indelible pen (sharpie) to mark the maximum angle it turns, then wiggle it back and forth vigorously a few hundred times and see if it is improving. If so - continue, if not - stop.

The black tap handle is actually removable, it just pulls of, and reveals that it is just a glorified hex-key. Could I just use a longer hex key to provide better leverage

I would try that. but I wouldn't extend the leverage very far.

or is there a problem in applying too much force?

Most ball valves seem to be pretty robust but I'm sure you could destroy one (or the surrounding pipework) with a 6-foot lever.

Should I also grip the pipes around to ensure they aren't put under force?

Yes. I would try to do that when you start to apply greater forces than normal. Maybe it is possible to firmly clamp the valve body to a strong length of wood or steel and hold that when applying leverage to the valve handle (or hex key)

tap gently with a hammer as I turn the tap

Worth trying even if there is a 1:100 chance of having any effect.

use spray oil (e.g. WD40) and let that soak in

I can't imagine that would help, but again, it is a cheap and easy option to try, and less likely to cause damage than the other ideas you listed. So if you have lots of time, this is one to try first.

If all fails, I would rent or buy a pipe-freezer and just replace the valve. You need a plan B immediately ready to hand in case this goes badly wrong. My plan B would be to turn off the water to the building using the valve in the street but that option might not be available to you.

A- This is not an "overflow" pipe.

B- It is not connected to an "overflow" valve.

C- It is a drain pipe connected to a Temperature & Pressure Relief Valve which is not only a legal requirement, but a critical safety device to prevent an over-pressure event (boiler explosion- google it)

For years and years, it was common to see no pipe at all. The pipe is a great idea whether or not it is plumbed to a drain or alternative location because if the valve activates it is less of a hazard to humans when scalding steam and water are directed downward and away from hapless victims. Some (if not all) localities require that they be plumbed to the exterior of the building or to a floor drain if the appliance is located in the home. Appliances outside of the home (as yours appears to be) may not be subject to the same requirement. Check with your local building code authority office.

P.S.- there should not be water coming from that drain pipe. If there is you may have a defective TP Valve (or worse, a malfunctioning boiler). Call a plumber.