Is exposed Roxul insulation safe

Rather than go to the trouble and expense of tying external insulation from below ground to the siding, I'd let them stop with the below ground insulation. It at least will add a little extra R value.

Then, since your basement is unfinished, but you do use it, I'd go ahead and add additional insulation on the inside of the house* - from the ceiling on down, at least to several feet below the external ground level if not all the way down. (Those extra few feet won't do much on the bottom but they will help some and they won't cost much more...)

*Making sure that any other potential water issues have been dealt with

The key thing with insulation is to get a continuous thermal break. Even a small gap in the insulation can allow huge amounts of energy loss, and since you are running heat downstairs, while I don't know precisely how to calculate the long term savings, there will be one, and more importantly, your basement will be much more comfortable for year round use.

I wouldn't attempt to remove the existing insulation unless there's a health hazard (e.g. mold). It was put there during the middle of framing because of how difficult the space is to access. My personal solution would be to spray a little big gap filler in there because it's designed to expand and provide an airtight seal. Other types of insulation, like fiberglass and Roxul lose their effectiveness when compressed. Much of the insulation in them comes from trapping air in pockets. So you'd get less efficiency after cramming it through a small crack.