[RPG] What happens when ability scores reach 0

Yes. This is correct.

To answer you bullet for bullet

1. Yes. This is what these do.

2. Yes, this is a correct reading, if you go from 18 to 20, you can still increase to 22.

3. Yes*, most things in 5e stack, though sometimes things don't stack with themselves (this isn't a bonus though, it's an increase, it's different). However, this is probably rare enough that this should not be an issue in a campaign (an if a DM gives out more than one of these per PC it's their own fault if their game breaks because of it). *I don't have this item's text in front of me so this is barring a caveat in the description, which if it existed would likely preclude this particular bullet anyways.

So basically, yes. This is an item (among several and a L20 class feature or two) that breaks the normal rule for maximum 20 ability scores. It breaks it, and it does so permanently for the character.

Pathfinder is pretty straight forward when it comes to boni and penalties, and we hage a Word of God:

A penalty DOES effectively lower an ability score, but it's temporary. When we have an effect that does damage or drain, we're intending that effect to last as long as it takes to heal the damage by outside sources. When an ability score reducing effect has a built in duration, after which the reduction goes away, we use "ability score penalties" instead of damage.

James Jacobs, Creative Director [at Paizo]

Therefore:

1. Take the Ability Score as basic number
2. Apply all relevant permanent modifieres, like enchantment bonus and drain, this is the Modified Ability Score, it will determine our 'Base' from where to modify further.
   1. If you are at an ability of 0 or negative: Roll a new character...
3. Sum up all temporary: boni, then damage and finally penalties (as negative numbers), one after another, apply the following on the way:
   1. Should drain + damage get you down to a score of 0, nasty things happen (unconscious, crippled, etc)
   2. Should a penalty try to lower the number to 0 or below, set it to 1.
4. Unless you hit a branching case (which is either dead, icapicitated or a score of 1 with bonus of -5), calculate the current Ability Score Bonus from the Modified Ability Score:
   • round down the result of \$\frac{\text{Modified Ability Score} - 10}{2} + \frac{\text{Sum of Boni, Damage and Penalties}}{2}\$
   • However RAI from the FAQ seems to be more the recalculation of the Ability Modifier each time with each Score: \$\frac{\text{Modified Ability Score}+\text{Sum of Boni, Damage and Penalties} - 10}{2}\$

So, you have strength 10, no strength damage and drain. Now you apply no bonus and a penalty of 5. The resulting value is Strength 5. Strength 5 results in an Ability modifier of -3. Would you have Strength 11, the modifired strength would be 6, and the Ability Score Bonus thus -2.

Now, your other question is similar:

You have 5 Ability damage and a 5 Ability Penalty applying to the same Ability. Here it gets a bit tricky. First, we check the Base score again. 10. Apply the bonus (0), then drain (0), then damage (5) and finally a penalty (5). After the damage, the Ability Score is 5. Applying an ability penalty of 5 to an ability of 5 is not possible, as Ability Penalties may never drop a score to 0, but instead they drop it to 1 instead.

This is played straight as just tracking them and then stacking them, as seems to be the general voice on the paizo forums:

You track each [penalty] separately and they stack. ... They don't actually reduce your score. Ability drain on the other hand, however DOES actually reduce your score.

... that ray of enfeeblement couldn't have reduced your goblin's strength to 0 (seen here)

also

Penalty and Damage don't lower the score, only Drain. Because of that, your score stays the same for the purpose of required stats.

The reference to lowering ability scores below 1 only applies to temporary effects, such as those from Ray of Enfeeblement, but not from diseases (which last permanently or until removed). You can't be rendered unconscious or dead by a temporary penalty from a spell, but you can by a permanent penalty from a disease or curse. (seen here)

also

ability modifiers are a summed modifier, not a separate set of individual bonuses or penalties. Ability damage doesn't directly reduce the relevant stat but it follows a similar logic. By this I mean: gaining 4 ability damage is a -2 penalty, not two -1 penalties. These are mathematically distinct concepts in pathfinder rules. (seen here)

but:

[That the same spell can't stack with itself] comes from page 208 of the core rulebook:

"Spells that provide bonuses or penalties...usually do not stack with themselves." (seen here)

Now, your test cases...

• Score 10, 5 Penalty, 5 Damage

The sum of the reductions is 10, but some of it is a penalty. As Penalties can't reduce to less than 1, the score is 1.

• Score 10, 4 Bonus, 11 Damage

As long as the Bonus of 4 stays, your score is 3. As soon as it fades, you go to -1 with all consequences, till the damage is healed to some degree.

• Score 10, 4 Bonus, 11 Drain

Your Modified Ability Score is -1. You may never have an Ability Score that is -1 from permanent sources. Drain is permanent. You die instantly.

• Score 10, 4 Bonus, 5 Penalty, 9 Damage

Sums up to an effective 10 reduction of the score, BUT, there is still some Penalty (1 point!) of it. Penalties still can't reduce to less than 1, so score is 1.