History
According to Ben Robbins, David Wesely ran a Napoleonic wargame called Braunstein in 1967:
Major David Wesely took his usual wargaming group and tried something a little different. Instead of having them command armies he set down the two opposing leaders in a Prussian town before the battle, their troops nearby but not on stage. To give the other players something to do he let them control other people around town: the Mayor, a school Chancellor, some revolutionary students, etc. The humble town was the eponymous Braunstein, “brown stone” in German.
This style of play marked a subtle shift in gaming, from commanding units to controlling individual personalities. As Robbins says, he was "the very first GM." Wesely explains:
The idea of having an all-powerful Referee who would invent the scenario for the game (battle) of the evening, provide for hidden movement and deal with anything the players decided that they wanted to do was not taken from Kriegspeil but was mostly inspired by 'Strategos, The American Game of War', a training manual for US army wargames
One of the players in that game was Dave Arneson. Arneson truly understood how to play in this new type of game, using his imagination. When Wesely joined the Army Reserves, Arneson started running his own games, but set in a fantasy world instead of Prussia. He used the Chainmail rules to handle combat. Arneson called his world Blackmoor.
One thing led to another. Arneson met up with Gary Gygax and they exchanged ideas and rules. Dungeons and Dragons was the eventual result.
Impact
So to summarize why Braunstein is important to hobby gaming:
Braunstein was the first game to popularize the idea of one player = one character.
Players weren't limited to actions in a rule book. It allowed players great latitude to take creative actions that would be interpreted by a game master.
Dave Arneson, a player in the first Braunstein game, influenced Gary Gygax and helped write the first D&D rules.
Possibly, Braunstein ("brown stone") created a naming convention that was used by Arneson ("Blackmoor") and Gygax ("Greyhawk").
Interesting Tidbits
Ironically, Wesely did not like the term "role-playing game," instead preferring "adventure game."
Wesely shares credit for inventing the RPG. He says that Micheal J. Korns published Modern War in Miniature, "a set of miniature rules with all of the features of an RPG," in 1968. This was a simultaneous invention, since Wesley and Korns had never met at that time.
Wesely claims to have invented polyhedral dice for gaming.
The premise of your question is somewhat incorrect. Level 20 is a standard progression limit only in 3e and 5e.
In First Edition AD&D, there is no level limit. Specific class advancement tables describe advancement from anywhere from 29 levels (cleric) to 9 levels (fighter) but only for purposes of showing how high certain abilities can go, they all note you can go on up infinitely from there. The big differentiation is "name level," which is usually in the 9-10 range, where the character stops getting as large level advances in hp and starts focusing on building kingdoms and whatnot.
In BECMI D&D (Red Box Basic), you can go up to level 36, and there are level breaks from Basic (up to level 3), Expert (up to level 14), Companion (up to level 25), Masters (up to level 36), and Immortal (past that, cashing in XP for power) with differences in those levels of play.
In Second Edition AD&D, advancement is described on convenient charts up to level 20 but there is no limit, with a breakpoint at level 9-10 where you stop getting full hit points with each level. It has a section in the DMG about how play gets harder to be satisfying at higher levels and that you probably need to shift campaign styles. In terms of playstyle recommendation past 20 it says
Consummate skill and creativity are required to construct adventures for extremely powerful characters (at least adventures that consist of more than just throwing bigger and bigger monsters at the nearly unbeatable party). Very high level player characters have so few limitations that every threat must be directed against the same weaknesses. And there are only so many times a DM can kidnap friends and family, steal spell books, or exile powerful lords before it becomes old hat.
It then recommends retirement as an endgame.
In Third Edition D&D, advancement is described up through level 20, with levels past that described in "an upcoming rulebook." It was lightly treated in the DMG but then more fully in the Epic Level Handbook in the 3.5e days. 3.5e and Pathfinder are lightly changed derivatives of 3e by design and so aren't really different editions with different ideas driving them as far as this goes. The Epic Level Handbook describes its intention, which is to change playstyle from the level 1-20 model to being legendary, allowing PCs to "wield powers that other characters (even 20th-level ones) can only dream about." It notes that PCs may have had the thrills of running nations and political machinations come and go and this is their gateway to discovering the secrets of the universe, plugging into the primal cosmic battles, etc.
You may also want to review the 3.5e DMG's discussion on epic characters and why attack and save bonuses cap out at 20 on p.207. (Summary: too many attacks causes slog and too much disparity between faster and slower save and BAB progressions causes balance issues). Also on p.210 they explain that many classes have been balanced assuming that 1-20 progression and that balancing classes for infinite progression is way harder.
In Fourth Edition D&D, the level limit is 30. There is no implied "soft cap" at 20. Play is grouped into rough "tiers" from the heroic (1-10) to paragon (11-20) to epic (21-30), but it is a continuum.
In Fifth Edition D&D, the game describes four tiers of play (1-4, 5-10, 11-16, 17-20) with "epic boons" available after level 20.
From this, you can take away several things.
The nature of play changes with level. Kicking down the door and killing something works well as a low level adventure and less well as a high level adventure, due to both repetition and the powers and abilities available to higher level PCs and foes, so shifting to more political or grander-scale adventures becomes desirable. Some editions formalize this with tiers, others say "low level, high level, and very high level", etc. What level that is varies by the specific D&D edition and its core rules.
There are a variety of "soft caps" and "hard caps" across the editions - tier boundaries, name level, etc. Only in 3e and 5e is 20 specifically a meaningful number that one might describe as a levelling cap (with later progression options). These numbers are not based on some arcane math but on when the designers feel like gameplay breaks down under its prior level paradigm. 4e is reusing the "epic" term but there is a continuum from 1-30 where epic can't be considered a meaningful cap, even a soft one, it's a breakpoint like the one at level 10. It's basically just using previous edition words for that level band to comfort people.
"Epic Level" play is a 3e thing based on a very specific product and terminology coined for 3e. Most references you've seen to "post-20 play" and "epic" are just an outgrowth of 3e play specifically. You are seeing something "across all of D&D" which isn't really across all of D&D.
Since versions of D&D mostly share certain rule similarities, the breakpoints of power - mostly cemented by what spells become available (fly, teleport, wish, etc. change the dynamics of the game by their availability) tend to be in around the same spots. So short of devising new things (tenth level spells, epic powers, etc.) versions of D&D that use the traditional spell advancement of "a new spell level every couple character levels" cap out spell power right before 20, where then it becomes a game of "more, but not really different" without additional rules that are pointless to include in core books where 99% of people don't ever get up to level 20 anyway. But this means that the around-level-20 breakpoint isn't really deliberately designed, it's more of an inevitable endgame of the spell system, unless you deviate from it (as 4e did). Even in BECMI, the Master rules (level 26-36) are only 32 pages long and are basically some new spells and then siege engine rules. When I played Basic no one ever went past Companion because the game got pretty weird and uninspired there.
Given a class-and-level system of D&D's kind, and the kind of Vancian casting powers traditionally available at levels around 5 (fly, fireball), 10 (teleport, raise dead), 18 (wish, miracle) then you get a similar need to change playstyles at those milestones, with 1-5 being your gritty stuff, 6-10 being (super)heroic, above that needing to change more to political and larger scale concerns to keep challenge and interest, and around 20 becoming a point of diminishing returns where you need to do something different to maintain challenge and interest given how spells etc. cap out there. BECMI Master pushed this past to 36 and got boring for that whole range; 4e went to 30 by discarding the Vancian tradition other editions share.
So it is incorrect to say that 20 is a soft cap across most editions, but this is the reason behind it in 3e/5e and the other "caps" and "breakpoints" and "tiers" in other editions in general. It's an emergent condition of the kind of ruleset D&D is and its historical trappings (Vancian magic being the most important) driving a change in playstyles at certain power inflection points. The designers explicitly talk about this in each edition's books regarding high/various level play.
If it needs to be stated more simply, 20 is not a magic number, it's just when having 9th level spells gets old.
Best Answer
The official thread for E6 references the idea that 6th is the beginning of the “heroic fantasy quartile” as defined by discussions prior to the development of E6, but this isn’t really the answer because: 1. when answering “why 6th?” this isn’t the answer given, and 2. the entire quartile premise isn’t really accurate. Rather, the designer’s stated reasoning is purely empirical: in their experience, 6th worked best. They don’t specifically offer speculation as to why 6th worked out well, they just tried different things and found 6th most comfortable.
However, there are many reasons why that happened, that we can see from a system-design standpoint. 6th level has a lot of notable advantages to it.
Generally, more levels means more toys to play with, more options available, more customization possible. Removing levels from the game thus comes at a cost. So balancing “having enough levels to have a lot of options” against “having few enough levels to avoid the problems of high levels” is a two-way street.
In specifics, the biggest single reason is the first iterative attack gained at +6/+1 BAB. Since that is achievable, it gives a rather nice benefit to taking only full-BAB classes—6th is often one of the best levels for such characters. If you never got to 6th, you wouldn’t “lose out” nearly so much if you missed a BAB or two.
6th also bumps both good and poor saves, avoiding any awkward fractions that go unused.
You can take your first level in many prestige classes at 6th. Getting to 6th thus allows prestige classes to exist in the game, where otherwise they mostly wouldn’t.
As I have discussed, the difference between 3rd and 4th level spells is vast, far more than between 1st and 2nd or 2nd and 3rd. The manageable potency of 3rd-level spells, versus not having spells at all, is a significant factor. While this is kind of the inverse of “why not higher than 6th?” it is relevant because the problems that limiting things to 6th fix are much less severe below 6th. Going lower than 6th thus solves fewer problems than stopping at 6th does.
This list should not be taken as exhaustive, but I do think these are the biggest reasons for not lower than 6th.
There is also another thing to point out: the game changes massively between 1st and 2nd level. It’s almost like playing a different game entirely. At 1st level, the game is hyper-lethal—many characters are downed by a single attack, and almost no one can survive a critical hit. Wealth is massively limited, usually (a lot) less than 200 gp, while at 2nd the expected wealth is 900 gp. A lot of classes (e.g. paladin, ranger) don’t even have their signature ability yet, since the need to avoid overpowering single-level multiclass dips shifts some signature features to 2nd level instead of 1st. Since 1st level is such a completely different game, staying that low brings with it a large array of changes—you’re basically not playing the game that 3.5 or PF represents at any other level.