I assume your gap is on the indoors side?
Regardless, if air is filtering through into the living space, there must be a gap outdoors ( external chimney) as well as indoors.
I recommend you check out the caulking along the chimney to the exterior siding as well as addressing the interior gap. Since this gap inside is directly adjacent to wood trim, I also have to assume that excessive heat is not a factor. Is the metal you are referring to an insert trim of some type? A pic would be a big help here.
If the assumption are right and the gap is 3/4 inch wide, I would use a caulking backer in the gap before using a good grade painter's or acrylic caulk as a topper. The caulking backer is a round foam solid tube that can be cut to length, inserted into the gap to seal it and hold the caulk. After the caulk has cured, it can be painted to match the trim. If for some reason you are concerned about high heat in this area, there is a fire rated caulk made especially for gaps in fire or common walls. Readily available at any box or hardware store from a number of manufactures.
Ok after reading your comments, and learning that the black vertical sides are slate, I would still do similar to above. Perhaps after injecting larger amounts of fire rated caulk as deep into void as possible, then use the solid, tube style caulking backer to fill the majority of the larger, deeper gaps, followed by using some paint-able silicon/acrylic mix caulk.
Before you apply the finish caulk, run a nice straight line of painter's tape up the front of the slate even with the edge of the wood trim. When you fill over the foam backer with the caulk, use a small putty knife to draw the caulk smooth and flat. Immediately remove the tape while caulk is fresh. This should result in a nice crisp straight line. Let the caulk cure for 24 hours. If there is some shrinking or cracks, no problem, simply repeat the process with new tape and another coat of caulk.
Alternately, you could also use a setting type drywall mud, such as durabond 30 or 45 to fill the deeper gaps etc. Then again use a second coat to create a smooth straight extension of the wood trim to the slate.
Either method will result in a neat, easy to paint fill that should stop your air infiltration problem and make the finish look nice.
The way fire insert stoves work is relatively simple in conversion.
You cut the damper out of the fireplace, run stainless steel pipe (6" dia.) up the inside of the chimney. The lengths are held together with 3-6 sheetmetal screws so this is an actual inside pipe assembly and it hangs off a sheetmetal cap that covers the top of the chimney and is silicone caulk sealed to the top of the chimney flue tile so it's air tight to prevent exactly the type of heat loss you mention. The pipe that extends above the sheetmetal sealing cap gets a standard stovepipe cap to prevent downdrafts and water intrusion from precipitation.
The bottom gets a piece of flex stainless pipe that joins the chimney stack to the stove.
The advantage of this is you are no longer at the mercy of cracked flue tiles and the fire hazard that represents. The 6 inch pipe runs at a higher temperature and tends to not creosote up. There are standard round chimney brushes for these so the soot that accumulates near the top is easily removable by popping the cap off and running the brush and 3-4 rods down the length of pipe.
Depending on where you live and your county's rules, this conversion may require a permit, but it saves you a lot of lost heat and drastically lowers your liability from flue fires.
The stainless pipe will need inspection periodically for pinholes and monitoring for soot buildup and a good brushdown every so often.
Best Answer
I have used rock wool around stove and fireplace inserts to close up air leaks, it is about the best in my opinion for this some stoves use it in the fire box above heat pipes to channel the heat and it lasts quite a while in there.