I have a 3-contact switch like this. 
As you can see, the switch doesn't have an OFF position, it diverts the current either between middle and left, or middle and right contacts.
Had to attach it to a wall lamp that didn't have a switch, by drilling a hole in the base of the lamp:
I have therefore one contact that I am not going to use and where I don't want current to flow at all. I cut it off and rolled all over the switch body in the place with 2 layers of insulating tape. The remaining 2 contacts were rolled in a layer of insulating tape as well, after connected to incoming live (middle contact) and lamp live (left contact) wires respectively.
As I drilled the hole in the base of the lamp, it turned out that the side of the switch with the spare contact may be touching the drywall a bit.
The question: since the current is supposedly trying to go from middle contact to the spare contact when lamp is turned off — is it liable to cause problems such as heating up its insulation or similar — simply by being live?
I am grateful and have upvoted all the three current answers, each of which provides relevant points for me. I feel that all three taken together answer the original question practically fully, but since I can only accept one, I gather I should accept the one, which most directly addresses the original question.
Best Answer
The lamp appears to be made from metal. You need at least two insulation barriers, or protective earth, for this to be anywhere near safe or legal. The protection mechanism - and implications are slightly different.
Protective earthing means that the chassis of the lamp has a robust connection to PE of the electrical wiring, so any fault current will pass trough the PE conductor, with low enough resistance that no dangerous fault voltage will be present on the apparatus. It may additionally trigger ground fault devices, if such are fitted in the system.
Double insulation means that the chassis is not grounded, but you have two protective barriers between the user and the live current. If either one fails, there is no risk of electrocution. One device in this instance may be approved heat shrink tube over the contact terminals. The other insulation barrier may be secure fitting of the switch in such a way that it will not touch the chassis.
Electrical tape is for temporary fixes. It's not a permanent solution.