It's difficult to model the situation with rational analysis, there's too many intangible factors. You could do an empirical test. You need to support 20 lbs per fastener. We can apply a safety factor of 3 for ultimate strength, so the fastener should support 60 lbs without actually breaking. So you would need 2-4 fasteners to support your weight. Round down to the closest whole number. Install the clips as you did in the wall, except now install a metal strap between the screw head and clip. Arrange the straps so you can step into them to weight the system. Arrange the straps such that your weight is distributed evenly to each fastener.
Weight the system and see if they break. If you live in a seismic area, bounce on them a bit and see if they break. You'll either be able to sleep better or you'll know what to do next, depending on the outcome. Obviously there are better ways to set up an empirical test, I chose to illustrate a quick and dirty method just as an example. Be sure you are protected from flying shards of metal.
Regarding an increaser for the number of fasteners. No, you can't do that. It is a valid concept though, for example you can use a higher allowable bending stress in multiple floor joists than you can in a single use situation such as a header. The concept is not generally applied to fasteners.
Response to OP's Update
Shear strength in relation to fasteners partly depends on what the fastener is holding. In this case it's known as a metal side plate condition, meaning the expected failure mode will either be the top of the screw failing through the shank (shear) or the wood collapsing under the compression from the screw. It's rare in reality to have a perfect shear condition, there is usually some bending and tension components as well.
A true shear condition would something like a metal strap screwed to the wood surface and all the force was parallel to the wood surface, exactly perpendicular to the screw shank. In your test, you mostly have the vertical shear component, but there is a tension component as the center of mass is away from the wall surface. We can safely ignore the tension component in calculating a working load since 80# in pure shear is more conservative than 80# shear and, oh... say 15# tension combined.
A picture of the clip was helpful, I imagined a much worse condition. Either way, the ultimate strength will not be proportional to shear alone, there are other factors difficult to model, thus testing is the best approach. The failure mode you experienced is a bending failure, but your actual installation, while having a bending component, is in fact mostly a shear condition.
The duration of load is a factor. The usual allowable stresses specified in construction are for permanently applied loads. The allowable stresses can be increased for shorter durations, 15% for a few months, 25% for a few weeks, 33% for a few minutes. Meaning we should reduce the allowable load determined through short term tests accordingly. But we also don't know the ultimate load since you didn't achieve failure. Just as well, uncontrolled destructive testing can be a little too exciting. You also haven't run multiple tests (I assume) to confirm you are getting consistent results.
Let's say you did run multiple tests and they all actually failed at 80#. When you apply the 3x safety factor, then adjust for duration of load, you end up with a working load of 20#, exactly what you need. Considering there was no failure experienced, and the installation does appear to be predominantly shear, I think your installation is safe. Barely. Next time around, use heavy ordinary wood screws ;)
This looks like a bad piece of engineering design to use a slip finger bracket to mount a cantilever paper holder. This type of bracket would work pretty good for a towel bar that was mounted on both ends or possibly for a soap dish that was mounted right on center.
I can suggest two possible courses of action that would solve the problem as long as the finger bracket on the wall was very nice and secure.
You could mix up some good quality epoxy (the kind that takes 24
hours to set up) and force it into the finger slots on the bracket
and on the fingers of the wall bracket. Then install the roll hanger
and let the epoxy set up. Note that this is a pretty permanent fix
and future removal would require cutting the hanger off with a dremel
tool or some such.
Alternatively you could clean up the wall and hanger assembly to
remove any oil and soap film and then re-install the hanger. Rubbing
alcohol can work great for this. Then run a small bead of clear
silicon sealer all around the perimeter of the bracket. This
sealer will take the twisting load off the fingers and transfer it to
the tile which should stop your problem. For removal the silicon
sealer can be cut away with a razor blade.
If it is the finger bracket on the wall that comes loose then you'll have to investigate how it is currently mounted. If the two screws there are already go directly into a stud or wooden cross member then some longer screws may be in order. If the screws just go into some cheap plastic hollow wall anchors then you may want to investigate some more secure types of metal anchors such as toggle bolts or molly screws.
Best Answer
The screw you're showing is a self-tapping metal screw. It's designed to cut it's own channel into metal and grab hold by cutting it's own threads - but it's not meant for repeated uses because repeat uses can damage it's own threads, making its hole too big for it.
Take it to the local hardware store and get one that is slightly wider in diameter, and use it in place of the old one.