The premise of your question is somewhat incorrect. Level 20 is a standard progression limit only in 3e and 5e.
In First Edition AD&D, there is no level limit. Specific class advancement tables describe advancement from anywhere from 29 levels (cleric) to 9 levels (fighter) but only for purposes of showing how high certain abilities can go, they all note you can go on up infinitely from there. The big differentiation is "name level," which is usually in the 9-10 range, where the character stops getting as large level advances in hp and starts focusing on building kingdoms and whatnot.
In BECMI D&D (Red Box Basic), you can go up to level 36, and there are level breaks from Basic (up to level 3), Expert (up to level 14), Companion (up to level 25), Masters (up to level 36), and Immortal (past that, cashing in XP for power) with differences in those levels of play.
In Second Edition AD&D, advancement is described on convenient charts up to level 20 but there is no limit, with a breakpoint at level 9-10 where you stop getting full hit points with each level. It has a section in the DMG about how play gets harder to be satisfying at higher levels and that you probably need to shift campaign styles. In terms of playstyle recommendation past 20 it says
Consummate skill and creativity are required to construct adventures for extremely powerful characters (at least adventures that consist of more than just throwing bigger and bigger monsters at the nearly unbeatable party). Very high level player characters have so few limitations that every threat must be directed against the same weaknesses. And there are only so many times a DM can kidnap friends and family, steal spell books, or exile powerful lords before it becomes old hat.
It then recommends retirement as an endgame.
In Third Edition D&D, advancement is described up through level 20, with levels past that described in "an upcoming rulebook." It was lightly treated in the DMG but then more fully in the Epic Level Handbook in the 3.5e days. 3.5e and Pathfinder are lightly changed derivatives of 3e by design and so aren't really different editions with different ideas driving them as far as this goes. The Epic Level Handbook describes its intention, which is to change playstyle from the level 1-20 model to being legendary, allowing PCs to "wield powers that other characters (even 20th-level ones) can only dream about." It notes that PCs may have had the thrills of running nations and political machinations come and go and this is their gateway to discovering the secrets of the universe, plugging into the primal cosmic battles, etc.
You may also want to review the 3.5e DMG's discussion on epic characters and why attack and save bonuses cap out at 20 on p.207. (Summary: too many attacks causes slog and too much disparity between faster and slower save and BAB progressions causes balance issues). Also on p.210 they explain that many classes have been balanced assuming that 1-20 progression and that balancing classes for infinite progression is way harder.
In Fourth Edition D&D, the level limit is 30. There is no implied "soft cap" at 20. Play is grouped into rough "tiers" from the heroic (1-10) to paragon (11-20) to epic (21-30), but it is a continuum.
In Fifth Edition D&D, the game describes four tiers of play (1-4, 5-10, 11-16, 17-20) with "epic boons" available after level 20.
From this, you can take away several things.
The nature of play changes with level. Kicking down the door and killing something works well as a low level adventure and less well as a high level adventure, due to both repetition and the powers and abilities available to higher level PCs and foes, so shifting to more political or grander-scale adventures becomes desirable. Some editions formalize this with tiers, others say "low level, high level, and very high level", etc. What level that is varies by the specific D&D edition and its core rules.
There are a variety of "soft caps" and "hard caps" across the editions - tier boundaries, name level, etc. Only in 3e and 5e is 20 specifically a meaningful number that one might describe as a levelling cap (with later progression options). These numbers are not based on some arcane math but on when the designers feel like gameplay breaks down under its prior level paradigm. 4e is reusing the "epic" term but there is a continuum from 1-30 where epic can't be considered a meaningful cap, even a soft one, it's a breakpoint like the one at level 10. It's basically just using previous edition words for that level band to comfort people.
"Epic Level" play is a 3e thing based on a very specific product and terminology coined for 3e. Most references you've seen to "post-20 play" and "epic" are just an outgrowth of 3e play specifically. You are seeing something "across all of D&D" which isn't really across all of D&D.
Since versions of D&D mostly share certain rule similarities, the breakpoints of power - mostly cemented by what spells become available (fly, teleport, wish, etc. change the dynamics of the game by their availability) tend to be in around the same spots. So short of devising new things (tenth level spells, epic powers, etc.) versions of D&D that use the traditional spell advancement of "a new spell level every couple character levels" cap out spell power right before 20, where then it becomes a game of "more, but not really different" without additional rules that are pointless to include in core books where 99% of people don't ever get up to level 20 anyway. But this means that the around-level-20 breakpoint isn't really deliberately designed, it's more of an inevitable endgame of the spell system, unless you deviate from it (as 4e did). Even in BECMI, the Master rules (level 26-36) are only 32 pages long and are basically some new spells and then siege engine rules. When I played Basic no one ever went past Companion because the game got pretty weird and uninspired there.
Given a class-and-level system of D&D's kind, and the kind of Vancian casting powers traditionally available at levels around 5 (fly, fireball), 10 (teleport, raise dead), 18 (wish, miracle) then you get a similar need to change playstyles at those milestones, with 1-5 being your gritty stuff, 6-10 being (super)heroic, above that needing to change more to political and larger scale concerns to keep challenge and interest, and around 20 becoming a point of diminishing returns where you need to do something different to maintain challenge and interest given how spells etc. cap out there. BECMI Master pushed this past to 36 and got boring for that whole range; 4e went to 30 by discarding the Vancian tradition other editions share.
So it is incorrect to say that 20 is a soft cap across most editions, but this is the reason behind it in 3e/5e and the other "caps" and "breakpoints" and "tiers" in other editions in general. It's an emergent condition of the kind of ruleset D&D is and its historical trappings (Vancian magic being the most important) driving a change in playstyles at certain power inflection points. The designers explicitly talk about this in each edition's books regarding high/various level play.
If it needs to be stated more simply, 20 is not a magic number, it's just when having 9th level spells gets old.
It was in the very first incarnation of D&D. Witness ye, the words of OD&D (Men & Magic) from 1974:
Magic-Users: Top level magic-users are perhaps the most powerful characters in
the game, but it is a long, hard road to the top, and to begin with they are weak,
so survival is often the question, unless fighters protect the low-level magical types
until they have worked up. The whole plethora of enchanted items lies at the
magic-users beck and call, save the arms and armor of the fighters (see, however,
Elves); Magic-Users may arm themselves with daggers only.
And therein lies your answer. A core part of D&D balance from the beginning has been that wizards wield incredible power but are fragile physically and can't use armor or most weapons. Gygax made that a part of the game, as it was how he envisioned his fantasy world grafted to a wargame to work.
The in-fiction justification back in the day for this restriction was simply that 'that class doesn't get around to learning that." Proficiencies as a formal idea that you could take instead of just having a monolithic bundle of abilities based on your class didn't come till later. If you dual or multi-classed, you could cast wizard spells in armor just fine by the way, there was originally no real inhibitor except that "wizards don't learn that in wizard school."
However, even "proficiency" is a compelling argument - armor's not "just clothing." Untrained people put on wetsuits, climbing harnesses, etc. in laughable, inefficient, binding, and frankly dangerous ways. Football players spend a lot of time micromanaging their pads and helmets and learning to move in them. The idea that "I'll just slap this armor on it'll be fine" falls down when its specific adjustment is what keeps you from getting bones broken from deflected blows, or from it getting caught on the battlefield/foes/weapons and dragging you to your doom.
One can also argue the influence of genre tropes (Gandalf didn't wear armor!) on this long-standing trope, but that's pretty much an opinion-fest, and is already on this SE as a closed question: Where does the stereotype that wizards can't wear armor come from?
Armor Across The Editions. According To The PHBs
0e, 1e, and 2e: Magic-users couldn't use armor because they weren't trained in its use, period. They are busy learning spells from books instead, and armor is a bit binding and impedes somatic components. Races that could multiclass or dual classing in general let you cast magic-user spells in armor.
3e, 3.5e, and Pathfinder: Magic-users can gain proficiency in armor but even then there's a spell failure chance for spells with somatic components because of armor's restrictive nature.
4e, 5e: Armor has no specific effect on spellcasting, though if you're not proficient you take various penalties to everything including spellcasting.
As you can see, the approach has really been quite consistent. Even before there were proficiencies, and after, the general explanation is "if you aren't proficient in armor, then you will have trouble with your spells," though that penalty has lessened over the years. It's a mix of game balance and realism - the same reason a wizard doesn't know armor and weapons is the reason a warrior doesn't know spells - in life, you have to make choices about what you learn, and "all of it" is not a feasible answer, at least not as a 16-year-old starting adventurer! In earlier editions it was harder to learn things in general as it was very class-based; now that there are proficiencies and stuff a wizard can learn armor like anyone else, by making that tradeoff to not learn something else useful.
Best Answer
In these three Legend Lore articles Mike Mearls talks about the fact a major design goal of D&D 5e is to Unite the editions.Specifically allowing the core game to be modified to play similarly to one of the past editions of D&D.
Uniting the Editions
While not all mechanics were carried forward from past editions, vancian magic was one of them. Vancian magic in a modified form with at-will cantrips, prepared spells, and rituals. The most direct answer to your question magic is what it is because that how it was presented in OD&D, AD&D 1st, AD&D 2nd, and D&D 3rd. It was modified in light of the experience with D&D 3e and D&D 4e and for reasons outlined in this article.
I could leave the answer like this but I feel it not complete. Some will wonder why Vancian magic in the first place?
It started, like in many mechanics in OD&D, with Chainmail. By the 2nd edition of Chainmail, wizards of varying power were introduced in the fantasy supplement. The four levels were Magician, Warlock, Sorceror, and the most powerful the Wizard. The difference between the different levels was not only in the power of their spells but the number of times per day they could cast spells.
When Gygax developed his Greyhawk Campaign he decided not to use Dave Arneson's system of magic reagents but rather was inspired by Jack Vance's Dying Earth series to create the familiar mechanics of spells in a spell book and the magic-users memorizing a limited selection of spells from the book.
This is corroborated in both Jon Peterson's Playing at the World and Kent David Kelly's Hawk & Moor series.
The mechanics are designed in 5e to reflect the spell memorization of classic editions of D&D which were inspired by the literature that Gary Gygax read most importantly Jack Vance's Dying Earth which were adapted from the Fantasy supplement of Chainmail which was used by Dave Arneson in his Blackmoor campaign.