Yes this can happen, especially if you don't remove the masking tape straight away.
It might indicate that the paint isn't properly adhered to the underlying surface, but it can happen even if you've prepared the surface properly (cleaned and dried).
The only real solution is to use something else - a straight edge held to the wall is always a good choice. However, that might not work well if the wall is uneven.
You could draw a line where you want to paint to and then hold a short piece of card up to this line as you paint in small sections.
From your update it sounds like when the previous paint was applied the surface wasn't prepared properly. Your best bet in those circumstances is to remove the top layers of paint - or even all of it - until you get a sound surface for you to paint on.
If we're talking a cast iron radiator, it's a good process.
First you need to remove the old finish. You can get it sandblasted, or you can get in there with some sandpaper for a few hours. A wire wheel on an angle grinder can get out a good bit, or you might have luck with chemicals to strip them. If you do use chemicals, you'll have to wait for them to totally dry, then sand the bare metal of the radiator just to make 100% sure all of the residue is gone and the surface is rough enough.
Next, you'll want to clean with a deglosser, to make sure all the sanding residue is gone, along with any other junk in there. The paint won't bond so well without it, and with something that expands and contracts as much as a heater.. you want to make sure it's thoroughly clean.
Now it's time to protect your walls and floors. Get a drop cloth, cut a slit in it and slide it over the unit. Tape it to the walls, tape it to the floor, and tape up as much of the slit you made as you can.
You'll then need to prime it, and you'll have to chose the primer based on the paint you end up with.
Time for paint. Oil based paint should hold up, but you'll probably have better luck with spray enamel paint. Use many thin coats rather than few thick coats. Make sure to read the specs on whatever paint you decide on, make sure it can withstand up to 220 degrees - Just to be safe.
Alternatively, maybe you can look into building a radiator cover.
Best Answer
Usually the second coat doesn't need tape because you don't need to get as close to the edge. But if you are going to, you'd want to re-mask for each coat. Otherwise the paint can seep under the masking tape while it's drying or it will pull non-masked paint off when you remove it.