Runes, as well as items like Sorcerer's Shoes and Haunting Guise offer FLAT penetration (i.e., you ignore up to X) of your opponent's MR.
The "average" champion has a base MR of 30, that doesn't increase with levels. With ~9 MR penetration from runes, sorcerer's shoes are enough to bring that down to basically zero.
The other options are reduction (triggered by champion abilities like Amumu, Ryze, or Fiddlesticks, or the Abyssal Scepter) which actually decrease your opponent's MR (and so stacks with penetration) and % based (the talent and the void staff). Of these, Reduction is the only way one can bring your target's MR below zero.
When a single character benefits from one or more of these, they are applied in this order:
- Flat Armor/MR Reduction
- % Based Armor/MR Reduction
- % Based Armor/MR Penetration
- Flat Armor/MR Penetration
Some of the above information can also be found in the game mechanics thread on the official LoL forums, here: http://www.leagueoflegends.com/board/showthread.php?t=3287
The answer to your question would then become:
54.36 MR * .85 = 46.206 MR - 8.55 = 37.656 effective MR. (I don't know the formula off hand to convert that to % damage reduced)
LoL has some inconsistent wordings, I'll agree.
It's rather simple, actually.
Consider a hypothetical scenario where Ashe attacks Garen (who is wearing Thornmail). Ashe has 200 attack damage and 30 MR (Ashe has no Magic Resistance items), and Garen has an effective armor of 100 (Because Ashe has armor penetration runes).
Case 1 - 100 Armor Garen, 30 MR Ashe
Ashe shoots an arrow, which would deal 200 damage. Thornmail uses this base damage to calculate the 30% retaliation. 30% of 200 is 60, so Ashe takes 46.15 damage (30 MR mitigates this to 77% of the base) from the Thornmail and deals 100 damage to Garen (100 armor reduces physical damage by 50%).
Case 2 - 200 Armor Garen, 30 MR Ashe
Garen wises up, and buys another 100 armor, and Ashe shoots him with another arrow. Ashe still takes 46.15 damage (same AD, same MR), but Garen only takes 66.67 damage.
Case 3 - 200 Armor Garen, 0 MR Ashe
But Garen is still not happy, so he buys a "Hypothetical 30 MR penetration item". Thornmail damage treats Garen as the origin, so it uses his magic penetration for the attack (same as proc items). The next time he runs into Ashe, Thornmail means Garen takes 66.67, and Ashe takes the full 60!
So in summary:
Thornmail Damage Return is based on the attacker's effective AD (can potentially be higher than just their AD for On-Hit attacks, like Mystic Shot or Parrrley, which will proc Thornmail), and the attacker's MR (reducible by the defender's magic pen).
The exact formula is pretty easy: Attack Damage * 30% * (1-MR%Reduction)
Best Answer
I think you're really asking two separate questions here. The first is, "What order do the various MR reduction / penetration effects follow?"
The answer to that is relatively simple.
So to answer the question as per the above, first Fiddlesticks' passive is applied to Ashe, reducing her Magic Resistance to 20. Next, the Abyssal Scepter effects is calculated, reducing Ashe's MR by 20, putting her at 0. Finally, the flat penetration from the Sorcerer's Shoes and the Haunting Guise are considered -- but Ashe is already at 0 MR, so there is no further MR to "penetrate".
Penetration can never bring a champion's defenses below zero.
So in the above scenario, spells would do 100% damage, so the full 100 damage in your example case.
To answer your second question, about the general case of negative magic resistances (possible on jungle creeps, and in some games featuring Karthas / Soraka / Kayle, etc), I bid you to look no further than this question.
The effects of negative defenses are exactly the same as the effects of positive defenses* - linear. In a nutshell, this means that an Abyssal Scepter will increase your spell DPS by the same amount regardless of whether the champion is starting with 100 MR or -100.
*the formula itself is different, because the armor calculation doesn't handle negative values well, but the end result is a continuation of Armor's linear scaling.