What you want is something like this:
![enter image description here](https://i.stack.imgur.com/f6HxQ.jpg)
Get some Red-Heads or other appropriate concrete bolt and fasten it down. Slip the 4x4 in and tighten it up.
Ideally, you should still have some posts properly embedded in the ground to help support the torque caused by someone leaning on the fence, but you can use these in between.
This is a tough one that will take a bit of work. From personal experience, any time you're going to be applying paint/primer/etc. to any kind of questionable masonry and mixed surfaces, it's going to be a good idea to clean/prep in several stages, then apply a masonry grade primer.
Depending on if there was any kind of debris, mildew, moss growing on the surface, a brush and/or pressure washer could be suggested. If the surfaces are clean and evenly covered, I would use a good stiff deck brush or similar.
I would also suggest a chemical cleaner/stripper containing muriatic acid, which will have a mild etching effect and ensure good adhesion with whatever you decide to paint with.
Then proceed with a good masonry primer, following manufacturers specifications. The same would apply for your paint.
In most of my jobs and personal projects, I would say a good paint job is 90% preparation, 9% good product and 1% skill.
In short, treat all surfaces the same, clean and prep evenly and somewhat indiscriminately. I would suggest a standard latex acrylic. Elastomeric paints tends to be heavier and really suited to new construction and initial application. If the house is older, it's most likely done most of its settling.
This reference is in regards to commercial new construction, but a lot of the premise will be the same.
http://www.masonrymagazine.com/5-08/cleaning.html
Good luck with the project.
Best Answer
Urethane or silicone caulk are the usual, but there are others that remain flexible. Read some labels at your hardware store. Flexibility is crucial, so don't use latex. If you have a lot of seasonal movement (frost heave), caulk might not be a good idea anyway.
Either match color or go with clear. Use as small a bead as possible and avoid smearing it out onto either surface by cutting a fine, angled point on the tube and pressing firmly.