Yes, that's correct. The calculations are:
Armour Resist: Damage Reduction = Total Armor / (100 + Total Armor)
Magic Resist: Damage Reduction = Total Magic Resistance / (100 + Total Magic Resistance)
As I mentioned in the comments, you're looking at the wrong aspect of armor. You shouldn't be looking at the percentage it reduces, but rather the amount of damage it actually prevents as a measure of how much longer that armor lets your survive.
Let's consider an example for a champion with 0 armor. Say he has 1000 health, just 'cause that's a nice round number. Now our champion gets caught in Garen's judgement, and starts taking 100 physical damage each second (at one hit per second). Clearly, our protagonist can take ten hits before dying.
But now, imagine our champion has the 50 armor you mention, meaning the incoming physical damage is reduced. Now (still with 1000 health) it takes 15 seconds for that bastard Garen to claim our champion's life. 50 armor extended our life by 5 seconds. (100 * (2/3) = 66.6 DPS; 1000 / 66.66... = 15 hits)
So our champion respawns, wises up, and buys 50 more armor. Now with 100 armor, we're sitting at a flat 50% damage reduction. Again we run into Garen, who's still dealing 100 base damage per second. It now takes him twice as long to kill us -- 20 seconds. Once again the armor has extended our life, again by 5 seconds. (100 * .5 = 50 DPS; 1000 / 50 = 20 hits)
Back in base, we buy another 50 armor. We're now at 150 armor, and 60% damage reduction. Enter Garen -- spinspinspinspinspinspin and it takes 25 seconds before our champion dies. Again, adding 50 armor extended our life by 5 seconds. (100 * .4 = 40 DPS; 1000 / 40 = 25 hits)
So even while the amount of damage reduced per point of armor diminishes (as you observed), the amount of time armor extends your life (or effective health) remains distinctly linear.
Best Answer
First we should correct this math, just for the sake of completeness. A character with 800 armor takes .11 damage for each 1 physical damage they would normally be dealt, which is 89% damage reduction. So to start with, if you have 100 AD you'll instead hit for 11 damage per shot.
So in the above example, what is the effect of a Bloodthirster versus a Last Whisper? Bloodthirster adds 80 AD, so that brings us to 180. How much difference does this make? 11% of 180 is about 20, so you spent 3500 gold to increase your damage by 9.
Last Whisper will reduce the target's armor against your physical damage to 520, as you said. To start with, this means that instead of taking 11% of damage dealt, he'll be taking about 16%. It will also increase your 100 AD character to 140, which means you'll be dealing about 22 damage per shot, or 11 more damage. Not a huge increase over Bloodthirster, but 800 is an absurd amount of armor.
Foregoing physical damage altogether with Wit's End is a reasonable option, especially since 800 armor means your opponent probably didn't build anything but Thornmails and Frozen Hearts. Increasing your damage by up to 42 ain't bad, and the defensive stats aren't awful, but consider that the rest of the enemy team likely isn't also building all armor, and will still get chunked by physical damage. Most AD carries don't deal a substantial amount of magic damage, so the MR shred isn't great, and now your abilities are less powerful against people whose builds aren't completely stupid and designed to target you specifically. Not to mention that if your team picked reasonably at all, you should have a good mix of damage, and poor Malphite with all armor and no health or MR will get absolutely destroyed by someone else while you go to town on the rest of their team.
Edit adding info from comments:
This Reddit thread details the difference in damage value between adding either Last Whisper or Blade of the Ruined King at varying AD, armor, and health levels. It found that LW will mainly be more effective than BotRK if (a) you build crit, (b) you have physical damage abilities, and/or (c) your opponent is building lots of armor, but not much health.