Shower – Mounting Framless Shower Door To Fibergalss Shower Insert

Your idea is pretty good; however, I'd be concerned about the load on the fiberglass wall.

Try to get a sideways piece of 2x6 or 2x8 between the studs. If the pipes are in the way, you'll have to notch them a little. (Probably the easiest method is to set a partial depth on a circular saw, cut a couple of kerfs and lever the waste out with a chisel or even a sturdy flat blade screwdriver.)

If the pipes are too close to the plane of the wall and you can't connect one side of the 2x8 to the stud, then you could run an upright partial stud (also on the flat) from the plate to the 2x8s.

the screw heads are starting to get chewed up

They are probably machine screws. I'd take one out and take it to a hardware store and buy a set with the same thread but with hex, star or square-drive heads. The length is not critical because you can always buy longer than needed and shorten them with a hacksaw (keeping a nut on the screw to straighten the thread with after cutting)

Is there a better solution

The shower door manufacturer may be able to supply replacement parts. I'd contact their customer support for advice.

Otherwise, after supporting the door securely, I would disassemble one hinge completely to see what is inside. Maybe there's a gasket of synthetic rubber that is supposed to grip the glass. It might be worth trying to supplement or replace that, for example, with a rectangle cut from a bicycle inner tube and maybe roughened a little.