That looks like a standard '3/8 compression x 1/2" MIP' (male iron pipe) connector. You can get it at any hardware store.
Normal procedure is to cut the tubing above the fitting and install a new one. Your tubing does not look too good, you have to have fairly smooth tubing surface for the new compression fitting to be able to slide on and seal. You may have to cut a little farther up on the tube. Be VERY careful not to kink or bend the tube while cutting it.
This is how compression fittings work.
You may want to purchase a mini tubing cutter.
http://www.youtube.com/watch?v=u77SJgIS4mY,
How can I get it to close fully?
The plastic handle is expendable. It will break long before the valve spindle. This is good. I would use an indelible pen (sharpie) to mark the maximum angle it turns, then wiggle it back and forth vigorously a few hundred times and see if it is improving. If so - continue, if not - stop.
The black tap handle is actually removable, it just pulls of, and reveals that it is just a glorified hex-key. Could I just use a longer hex key to provide better leverage
I would try that. but I wouldn't extend the leverage very far.
or is there a problem in applying too much force?
Most ball valves seem to be pretty robust but I'm sure you could destroy one (or the surrounding pipework) with a 6-foot lever.
Should I also grip the pipes around to ensure they aren't put under force?
Yes. I would try to do that when you start to apply greater forces than normal. Maybe it is possible to firmly clamp the valve body to a strong length of wood or steel and hold that when applying leverage to the valve handle (or hex key)
tap gently with a hammer as I turn the tap
Worth trying even if there is a 1:100 chance of having any effect.
use spray oil (e.g. WD40) and let that soak in
I can't imagine that would help, but again, it is a cheap and easy option to try, and less likely to cause damage than the other ideas you listed. So if you have lots of time, this is one to try first.
If all fails, I would rent or buy a pipe-freezer and just replace the valve. You need a plan B immediately ready to hand in case this goes badly wrong. My plan B would be to turn off the water to the building using the valve in the street but that option might not be available to you.
Best Answer
Right, so I sorted it. Will post solution here for anyone else who comes across this weird fitting!
Basically the threaded part is actually kind of a nut. So use a wrench to twist this anticlockwise, whilst trying not to ruin the thread!The cartridge then just pulls out.