League-of-legends – the formula for armor/magic resist reduction

As I mentioned in the comments, you're looking at the wrong aspect of armor. You shouldn't be looking at the percentage it reduces, but rather the amount of damage it actually prevents as a measure of how much longer that armor lets your survive.

Let's consider an example for a champion with 0 armor. Say he has 1000 health, just 'cause that's a nice round number. Now our champion gets caught in Garen's judgement, and starts taking 100 physical damage each second (at one hit per second). Clearly, our protagonist can take ten hits before dying.

But now, imagine our champion has the 50 armor you mention, meaning the incoming physical damage is reduced. Now (still with 1000 health) it takes 15 seconds for that bastard Garen to claim our champion's life. 50 armor extended our life by 5 seconds. (100 * (2/3) = 66.6 DPS; 1000 / 66.66... = 15 hits)

So our champion respawns, wises up, and buys 50 more armor. Now with 100 armor, we're sitting at a flat 50% damage reduction. Again we run into Garen, who's still dealing 100 base damage per second. It now takes him twice as long to kill us -- 20 seconds. Once again the armor has extended our life, again by 5 seconds. (100 * .5 = 50 DPS; 1000 / 50 = 20 hits)

Back in base, we buy another 50 armor. We're now at 150 armor, and 60% damage reduction. Enter Garen -- spinspinspinspinspinspin and it takes 25 seconds before our champion dies. Again, adding 50 armor extended our life by 5 seconds. (100 * .4 = 40 DPS; 1000 / 40 = 25 hits)

So even while the amount of damage reduced per point of armor diminishes (as you observed), the amount of time armor extends your life (or effective health) remains distinctly linear.

First, let's clear up a misconception.

Armor Penetration gives you a constant %damage bonus, regardless of how much armor the enemy has see here.

It doesn't matter if your enemy has 300 armor or 15 -- penetrating 10 of that will have the same effect on total DPS increase.

That said, let's take a look at why Armor Penetration is preferred, even with its linearity.

Attack Speed is easy to acquire via items.

Armor Penetration is not.

You can more than match the effects of attack speed runes with a single dagger - 420 gold.

The cheapest item with Armor Penetration is the Brutalizer at 1337 total gold. By this metric alone, Armor penetration runes are worth approximately three times that of attack speed runes.

It is easier to use items to stack Attack Speed over Armor Penetration -- but to get the highest DPS, you want to stack both.

Given that the amount of Armor Penetration available via items is strictly limited, if one's goal is to achieve maximal values for both Armor Penetration and Attack Speed, runes for Armor Penetration are much better than that of Attack Speed.