You've discovered something of a conundrum in current building codes. Modern houses are built to very tight standards, which means than exhaust vents commonly found in kitchens and bathrooms run the risk of depressurizing the house because not enough makeup air can infiltrate through gaps in the siding, doors, and windows, like it used to be able to do with older, leakier houses. Depressurizing the house is dangerous if you have any gas-burning appliances, because the pressure difference between the inside and the outside can cause products of combustion like carbon monoxide to get sucked into the house itself rather than through the vent that leads outside. Very dangerous. The obvious solution is to bring in a source of makeup air from the outside, which is what the builder of your house did.
...However, this introduces the exact problem you're encountering: there's a huge energy hit because (duh) the outside air is probably a very different temperature from the inside air! Building codes let you do this and a lot of ignorant builders will blindly follow the code, not realizing that they're basically undoing the effect of all the air-sealing work they're required to do by different sections of the code by punching a huge hole in the house that never closes.
There are a few things that can be done about this. Some are best done to be done during construction, while others can be retrofitted after the fact. Unfortunately, while installing an HRV or ERV will solve the problem, it will likely be cost-prohibitive because you will need to have the ducts redone in addition to buying the unit. Instead, I would advocate that you find a way to close up the hole when makeup air is not needed. The basic problem is not so much an uninsulated opening in your house but the hole itself. This hole allows active air movement, which in your case is undesirably cooling the house. So you need a method to close the hole when it's not strictly needed.
Option 1
The simplest possible solution is to just remove the makeup air grille entirely and seal up the hole in the wall on both sides, with insulation in between. This is probably perfectly safe if your house is made of wood and was built before 1995 or so; builders didn't really start to care about air sealing until well into the 2000s. For example, if you have an un-louvered bathroom vent, anytime your house starts to get negative pressure, makeup air will just come in through there. Your house is most likely leaky enough that makeup air will naturally flow through the many, many openings, cracks, and holes in your walls, ceiling, basement, etc. You might want to get a blower door test done to confirm this. If the result is more than about 8 air changes per hour at 50 pascals of pressure, you're probably just fine. Of course, if you put effort into sealing these cracks in the interest of energy efficiency, you're back to square 1 and you'll need external makeup air again.
Option 2
If your house is made of concrete blocks or you have a recently-built house made out of wood that was built nice and tight, that's not an option because you will have far fewer natural air leaks. A manual lever attached to a louvered grille would be the easiest solution and you can do it yourself very easily. You just open it when you turn on the range hood. The downside of course is that you'll need to remember to do this!
Option 3
A motorized damper that activates a louvered grille, as suggested by Philip Ngai, can also work, but it will probably have to be a custom job. I'm not aware of any products that do this in conjunction with a kitchen exhaust hood. That would be the most "set it and forget it" type of solution.
Option 4
There's one possible complication: If the makeup air vent is close to the exhaust vent, it's quite possible that using both at once will simply result in the exhausted air being pulled right back into the house through a big air loop. If this is your situation, none of these band-aid fixes will work; you will need to have one of the openings relocated and in that case you might as well install an HRV if you'll be re-doing the ducts and cutting holes in the house.
I own a mansard and we are battling this issue. We had our roof replaced last year and the roofer that did it didn't even install 1/10th of the recommended ventilation. Last winter was brutal in the north and we sustained substantial water damage from condensation inside of our home. We are working with another roofer to get the proper edge venting installed. Think of a cold glass on a hot summer day, that is what the inside of my roof decking looked like last winter.
Do not skimp or you will be sorry and sustain a substantially decreased lifespan of your roof, not to mention you could incur damage to the interior of your home if you live in the north (hot inside of the roof + extreme cold outside = large amounts of condensation).
Make sure you have a proper vapor barrier as well. Loose fill insulation = more required ventilation. I would also consider a power vent with both thermostat and a humidistat. I am not a professional, but I am a Mechanical Engineer by education and since our issue, I have researched proper ventilation of mansards extensively. Go with a very reputable roofer that will stand behind their work if you have condensation issues.
Best Answer
I am not aware of any venting requirements for a pantry - certainly not for any opening outside. If I were you I would close up the hole and seal it properly. For air exchange in the pantry, I'd put two vents in the pantry door - one low, one high, to allow air transfer.