In a perfect world, with perfect electronics, there would be no need for a grounding conductor. It's there to deal with imperfections. The most likely case for a ground fault in this installation would be if the black conductor lost some insulation and shorted itself against the frame of the fan. In such a situation, the casing could become electrified, and a grounded person, touching the fan body could receive a shock.
If it were properly grounded, you'd get a short circuit to the ground wire, and the breaker would trip.
However, the fan is mounted to the ceiling, and the possibility of someone touching it is low. (Unlike, for example, the frame of a fridge or stove, where human contact is part of normal operation.)
So, Is it safe? Quite likely. For added safety, turn the wall switch off when changing bulbs. Is it to code? No. -- Unless, as others have suggested in comments, the mounting box is grounded on the other side.
You can however, locate the breaker, and replace it with a Ground Fault Interrupt breaker, which would trip should a ground fault occur. This isn't something you should do yourself unless you have experience working in the panel.
Another possibility, if your switch happens to be chained off of an outlet, which is quite possible, depending on the routing of the wiring on that circuit, is to replace the outlet with a GFCI outlet, and wire the switch to the load side of it.
I hate to leave questionable connections behind walls. I dream of fire and destruction.
I would repair the tear unless you were certain that the wire insulation was not compromised.
The easiest way to do that is with an inline cable splice.
These are mean to be concealed in walls where repairs to existing wiring is needed, and are UL approved. I use them sparingly (they are a hidden connection) but sometimes they are necessary.
Yon need some slack, so you may have to open a bay next to the current one to get more cable. You can then run a new cable to the next box. In a pinch, you could use two splices and a short run of cable between them, but I would try to avoid that.
Best Answer
You are likely to have issues and I would strongly recommend replacing at least the LED driver box ("junction box"), if not the entire light.
The white cable that connects to the driver box contains at least two low-voltage wires, and is secured to the grounded metal driver box by a plastic collar. Contact between the metal box and the internal wires likely caused your issue, pulling on the cable probably caused further exposure of the internal wires, and if your wire nuts aren't well-secured inside the box due to some of this punching and pulling, you could have a real mess on your hands from low voltage + high voltage + ground contact.
Rather than punching holes in the ceiling, try to pull the light downward out of the hole and the driver box should follow.