How to seal between sill plate and foundation wall with wide gap

Here is a picture of the top of my uninsulated basement wall. The foundation is poured and from early 1970s. For orientation to the photo: the white in the lower left corner is the vertical face of the wall. The horizontal board is the sill plate. Attached to the sill plate is some mortar. In this photo it has separated from the top of the wall. (Edit: the picture happens to be in a location directly above a basement window, and the mortar has separated from the frame above the window. However, there are places along this wall where the mortar has indeed separated by a fraction of an inch from the poured foundation wall. An eighth to as much as a quarter of an inch, except above windows where it's more like 1/2")

I am in the process of insulating the rim joists. I'm using rigid foam board instead of spray foam, and I'm using caulk instead of Great Stuff / expanding spray foam. As part of the job I am trying to seal the gap between the sill plate and the foundation wall. In the places where the mortar has not separated from the wall, it's easy enough to just run caulk along between the mortar and the sill plate. However, where it has separated, I need a better solution. How can I do that?

If the gap between the sill plate and the wall was a little smaller, I could just run caulk in it. However, since it's maybe 1/2 an inch, would it be better to shove a narrow strip of wood into the insulation to give the caulk something to adhere to? Or I could cut a narrow strip of wood (say, 1" by 1/4") and staple that to the front of the sill plate in a fashion that covers up the insulation gap and then caulk along where that strip meets the foundation wall and also where it meets the sill plate.

Is there a better option you'd recommend?

That's my main question, but here are another couple of questions if you have interest or time.

Below is a stretch of wall where I removed the mortar that had separated from the top of the wall. You can see that the mortar on the right side of the photo has not separated from the top of the foundation wall. What the heck is the purpose of the nail that is sticking into the sill plate? It was buried in the mortar that I removed. You can see the head of another nail in the mortar in the image above. Seems weird to me…though I have never done this type of work.

What is the yellow insulation that is underneath the sill plate called? Why is the gap so tall? I would expect the sill plate to be basically right on top of the foundation wall with only a tiny gap.

And what is the purpose of the mortar along the top of the wall in the first place? Is it basically an attempt to stop air infiltration?

Edit: as follow-up to a comment I made, here's a picture of an anchor bolt in the sill plate.