As I mentioned in the comments, you're looking at the wrong aspect of armor. You shouldn't be looking at the percentage it reduces, but rather the amount of damage it actually prevents as a measure of how much longer that armor lets your survive.
Let's consider an example for a champion with 0 armor. Say he has 1000 health, just 'cause that's a nice round number. Now our champion gets caught in Garen's judgement, and starts taking 100 physical damage each second (at one hit per second). Clearly, our protagonist can take ten hits before dying.
But now, imagine our champion has the 50 armor you mention, meaning the incoming physical damage is reduced. Now (still with 1000 health) it takes 15 seconds for that bastard Garen to claim our champion's life. 50 armor extended our life by 5 seconds. (100 * (2/3) = 66.6 DPS; 1000 / 66.66... = 15 hits)
So our champion respawns, wises up, and buys 50 more armor. Now with 100 armor, we're sitting at a flat 50% damage reduction. Again we run into Garen, who's still dealing 100 base damage per second. It now takes him twice as long to kill us -- 20 seconds. Once again the armor has extended our life, again by 5 seconds. (100 * .5 = 50 DPS; 1000 / 50 = 20 hits)
Back in base, we buy another 50 armor. We're now at 150 armor, and 60% damage reduction. Enter Garen -- spinspinspinspinspinspin and it takes 25 seconds before our champion dies. Again, adding 50 armor extended our life by 5 seconds. (100 * .4 = 40 DPS; 1000 / 40 = 25 hits)
So even while the amount of damage reduced per point of armor diminishes (as you observed), the amount of time armor extends your life (or effective health) remains distinctly linear.
Okay, from my experiments, it goes:
Armor/MR reductions <--> Percentage Reductions
(They stack multiplicatively, so order is irrelevant)
Flat Reductions
This makes sense, since if flat reduction was the first thing applied, after armor, it may have failed to provide a benefit at all (since, for instance, the difference between 50 * 27% and 48 * 27% is minute at best. 50 and 27% being the base attack damage of Ezreal and the armor of Amumu, my two test characters.)
Shields act as health, and damage done to shields is likewise reduced by armor / magic resist. (Meaning shields on tanks can generally absorb more punishment than shields on squishies)
Best Answer
It is possible to get someone's magic resist/armor below zero with proper items/skills/spells. Like you said magic/armor penetration doesn't do that. Only % magic/ resist/armor reduction and flat magic resist/armor reduction.
MAGIC RESISTANCE REDUCTION
ITEMS
ABILITIES
Flat magic reduction:
Percentage magic reduction:
Trundle's Agony steals magic resistance of the target by 15% / 20% / 25%, this amount is doubled over the next 6 seconds.
Jayce's Transform: Mercury Cannon grants his next auto attack to reduce the enemy's magic resist by 10% / 15% / 20% / 25% for 5 seconds.
Karthus's Wall of Pain reduces the magic resistance of all targets affected by 15% for 5 seconds.
Kayle's Holy Fervor allows her autoattacks against enemy champions reduces their magic resistance by 3% for 5 seconds (effect stacks up to 5 times).
Nidalee's Bushwhack reduces the armor and magic resistance of enemies that step on the trap by 20% / 25% / 30% / 35% / 40% for 12 seconds.
SUMMONER SPELLS
ARMOR REDUCTION
ITEMS
ABILITIES
Percentage Reduction:
Flat Reduction:
Corki's Gatling Gun fires automatically every half second, reducing enemies armor with each hit by 2 / 4 / 6 / 8 / 10 for 2 seconds.
Heimerdinger's H-28G Evolution Turret, while in ranks 2, 3, 4 and 5, the turrets' attacks reduce armor and magic resist by 1 per hit stacking up to 50 times.
Kog'Maw's Caustic Spittle reduces the target's armor and magic resistance by 5 / 10 / 15 / 20 / 25 for 4 seconds.
Nasus's Spirit Fire reduces the armor of all enemies in the area by 20 / 25 / 30 / 35 / 40 for the time they are on it plus 1 second.
Rammus's Puncturing Taunt reduces the target's armor by 10 / 15 / 20 / 25 / 30 for 1 / 1.5 / 2 / 2.5 / 3 seconds.
Taric's Shatter reduces the armor of all enemies hit by its activation damage by 10 / 15 / 20 / 25 / 30 for 4 seconds.
SUMMONER SPELLS
SOURCE
LOL WIKI
EXAMPLE
An easy example that everyone can try out and find out if armor can actually go negative is this:
Heimerdinger's level 1 Armor is 10. So get Corki and get exhaust with the Summoner's Wrath mastery.
Go meet each other at a lane (level 1) and exhaust him and use Gatling Gun on him for 2 seconds and watch his Armor become 10 - 10 - 8 = -8!!