Use the core drill you mentioned before, make sure it is deep enough, say 4" to get to the bottom of the iron stub. That is the standard mounting depth.
To keep the drill in place, drill a hole in a piece of 3/4" plywood big enough (1'X4'??)to lay on the stair you need to drill with the hole in the place that the old post is located. Stand on this while you use the drill to cut the hole. Get another body on it too if you feel you need to, but if you can maintain good balance you will be fine. drill it down about 1/2" and set aside the plywood and finish the depth of the hole. you can always chip a little of the concrete out, to go deeper, but you will find that a well placed tap with a hammer and chisel or steel pin to the side will release the plug that you drilled.
The plywood will stay in place very well with the pressure of you standing on it, it does not take a whole lot of pressure to keep it from "dancing".
If you can use a water fed core drill.
Asking the basic questions is what gets us started.... not silly at all.
If they are as simple as nail holes from picture hanging and the like, and there are only a few of them, use spackle. It comes in small quantities, pints for example, so much will not go to waste. It is intended for only small holes.
If there are a lot of repairs, you may want to go with joint compound, it is pretty much the same as spackle, it is easy to sand, water based for easy clean up, but you will get a bunch more for the about the same money. It is intended for larger repairs, or new drywall install, can be built up, but it does shrink a bit, not so that it will fall out. If the shrinkage is noticeable, add another coat, no biggie.
Plaster is way overkill for this type of repair, it requires a skill to use. I have tried to use it before years ago, and it set up as soon as it touched the wall, quite surprising. I now know were it is to be used nowadays, and nail repair is not one of them.
Best Answer
A drill would be a good investment, lightweight general purpose cordless drills are available at a surprisingly reasonable cost.
The joint compound plan might work alright. You don't state how big the original hole was; if it is large (like, larger than the new thermostat) then you might have a tough time patching without something behind to "back up" the patch. Consider bridging the hole with a strip of stiff material, like a strip of drywall or even a wood paint stir-stick. You would slide the strip into and beyond the edge of the hole on one side, then pull it back so the strip overhangs both sides of the hole on the inside (I usually put a screw through the strip in the middle, to serve as a temporary handle). Run a screw through the wall and into/through the strip on both the right and the left, to pull the strip up against the back of the wall/hole on the inside, then remove your makeshift "handle". The attachment screws would have to be driven below the wall surface so you can patch the screw holes at the same time you fill the main hole with joint compound. Smooth and sand, then use plastic anchors to secure your new thermostat.