Concerned about the outcome of insulation fibers

Check the directions/specifications of your exhaust fan, it should say if it is rated to have insulation next to or on top of it.

It would probably be a good idea to seal the vapour barrier back around the fan.

All of your stated options are possible. However, unless this ceiling has absolutely no insulation, or you have a LOT of glass in this room, none of these options will really address the problem.

Put curtains on the windows and glass-paneled doors. Also, rent a FLIR camera, which takes a thermal image of what it's looking at, and take a reading of every surface in the warm room. The thermal picture will tell you if there is a significant rise in temperature anywhere, which indicates improper insulation, or conversely an unexpected DROP in temperature, which in the summer will indicate a leak in the vent. If you do see poorly insulated areas, you may be able to fill in the gaps with blown or spray-foam insulation, or you may have to drop the whole ceiling and reinsulate.

The most likely culprit for poor cooling in one space but not others, other than solar gain through a lot of windows or skylights (in which case curtains and blinds should give you a DRASTIC improvement), is inadequate air circulation into and through the space, or else poor insulation of the ductwork bringing air to this space. The easiest thing to check and fix is the filter; it should be replaced at least once a year and more like once every 6 months especially if you have pets. More than that, and the filter will become so clogged with dust it will impede airflow. The more surface area the filter has, the better, because the more dust it can trap before becoming clogged. I have a 16x25x4" electrostatic media filter and it works just fine for a single system cooling a two-story house. However, the filter will generally degrade the performance of the entire system.

The next easiest thing to diagnose is the condition of the ducts themselves. Take off a vent cover and inspect the inside of the ducts with a mirror; if there's a layer of dust, there's probably more of it closer to the main blower. Call an HVAC crew in for a duct cleaning. Again though, this will likely degrade the performance of the entire system, not just one room (unless there's a leak).

If the ducts themselves are clean, you may need to balance the system. Certain vents will get better airflow than others; it just happens, and which ones get the most airflow depends on the ductwork design. To direct more air downstream, or to underserved areas, partially close the vents in areas of the home that are getting plenty of air, which will direct more airflow to rooms that aren't getting enough. Also consider completely closing the vent closest to your thermostat, thus requiring the thermostat to get its reading from the air coming from other rooms; I've seen plenty of very poor designs where the thermostat is right underneath one of the closest vents to the blower, and so the A/C only runs for 5 minutes at a time because the vent blows cold air right onto the thermostat. This balancing process will require a little patience to get just right. Once set properly, the rooms in your home should each receive the right amount of air to maintain the desired temperature without rooms being too hot or cold.

Similar to balancing the system, understand that doors are very effective "valves" allowing air flow from the vents back to the air returns. If a room is too warm and you want it cooler, opening the door allows more airflow out of the room, which by the same token allows more airflow INTO the room through the A/C vent. Keep as many doors open through your house as possible, especially during the heat of the day. If this is not feasible, you can maintain privacy in your home while increasing airflow by adding a louvered panel to the door, which allows air to blow through.

The hardest thing to fix (impossible sometimes) but still easy to diagnose is the temperature of the air coming out of the ducts feeding the warm room. Get yourself an IR thermometer, and take a reading from each of the vents in the room or house. You should get a reading between 20-30 degrees cooler than the outside temperature. If the air coming out from ALL vents in the house is less than 20 degrees cooler than outside, the compressor is on its last legs, or the coolant is low, or there's condensation in the system. An HVAC tech can disgnose which of these is the problem in a matter of minutes.

If only a few rooms but not others have the problem, the ducts to those rooms are being exposed to outside temperatures, perhaps along an exterior wall or uninsulated ceiling. The fix is to somehow get better insulation to the duct, and you will likely have to tear out ceilings and walls to do this, unless the duct runs through an open attic space. Alternately, you may have a leak along the duct line leading to this vent, allowinfg most of the air to escape, and what's left isn't enough to cool the room. Again, unless the duct's in an accessible attic, you will probably have to tear out walls and ceilings to find and fix the problem.

The reason turning off the ceiling fan helps is that the fan is simply circulating the hot air in the vaulted ceiling above your head down into the room. This requires your A/C to cool the entire space (made more difficult by deficiencies in the ventilation of this room), when you really only need a 6-8ft "blanket" of cool air at the floor. By turning the fan off, you allow the air to "stratify"; cool below, warm above. If you have a nice "blanket" of cool air, and the fan is really high up in the ceiling, you can try extending the downrod of the fan to lower the blades into the cool air, so they grab less of the warm air from above. Keep the fan blades at least 7 feet off the floor though, and I would recommend more like 8-9'. Understand that, fan or not, the warm air will still mix with the cool air at a level determined by how much there is of warm and cold, and if that's at head height, you'll feel little relief even if the air around your feet is 60 degrees.