There sure is!
- Pick a size of a pool of d20 dice. The bigger the pool, the stronger the results.
- Next grab a d6, d10, a different colored d20, or even a coin.
- Roll the die pool and roll the extra die.
- If you got an even number on the die, pick the smallest roll from of the pool of die and use this as a result. Odd? You pick the largest die value from the pool.
With a pool of 4d20 I get the following results in Excel (using a total sampling of 10,000 rolls):
1 2 3 4 5 6 7 8 9 10
9% 8% 7% 6% 5% 4% 3% 3% 3% 2%
11 12 13 14 15 16 17 18 19 20
2% 3% 3% 3% 4% 5% 6% 6% 8% 9%
So roughly a 20% chance of critical success or critical fail, and a 4% combined chance of a 10 or 11.
Credit goes to sage for suggesting the even/odd roll instead of a coin.
And now some graphs, because everyone likes graphs, right?
Although technically this is bar-graph land, I included a line graph as well as I thought it made it easier to see in light of the number of bars that were being dealt with. I also increased the sampling to some 100,000 dice.
Not all role-playing game have fumbles. In fact, I suspect that were an intrepid soul to catalog every RPG—a daunting if not impossible task—, more RPGs would lack fumbles than possess them. However, many games have optional rules for fumbles for those players who like them, and many games will have specific elements that'll see a deeply flawed attempt yield consequences worse than mere failure despite lacking a general rule for fumbles. So, yeah, while the games you've played have had fumbles, fumbles are by no means a universal.
Possibly the original fumble mechanics
While I'm no role-playing game scholar, Chaosium's Runequest (1978) apparently included fumbles at least as early as 1980 (which is the version I extracted from my shelf, dusted off, and cracked the binding of when I flipped through it). Runequest (1980) has the following section:
Fumbles
An Adventurer using a weapon for which he has only a 5-20% chance of success has a 5% (a roll of 96-00 on D100) of fumbling. For every additional 20% capability an Adventurer has with a weapon, his chance of fumbling with it is reduced by 1%. However, an attack roll of 00 is always a fumble. Even an Adventurer with a 100% of hitting (actually 95% for dice rolls of 96-00 are always a miss) will fumble if a 00 is rolled. (20)
(It took a moment to find the page number: they're on the outside upper corner of each page. Pro Tip: If you're laying out a book, don't do that!)
A chart on which one is to roll if a fumble occurs appears soon after.
Jonn Rees apparently ran these numbers both for skill use and the chart. I'm looking at Steve Jackson Games' Murphy's Rules (1988), a collection of comic strips detailing amusing and incongruous game rules from the magazines Fantasy Gamer and Space Gamer (yes, there used to be several gaming magazines!), and John Rees submitted an oddity in 1984 that says
In a 30-minute Runequest battle involving 6,000 armored, experienced warriors using Great Axes, more than 150 men will decapitate themselves and another 600 will chop off their own arms or legs.
So, even then, while the sheer frequency of fumbles wasn't considered necessarily bad design, it was, at least, considered funny.
Speculation: Why fumbles exist
Fans of role-playing games may view the fictional reality the role-playing game creates as a heightened reality, which I'll call for convenience reality-plus.
In actual reality—the one most of us are in, like it or not—, very little that happens matters in the grand scheme of things. In reality-plus, however, everything that happens during the game matters, and spectacle is expected for good or ill. And spectacle during even everyday tasks can lead to comedy or tragedy that is a hallmark of reality-plus.
For example, while training with his Great Axe [sic], Argath of Sartar chops off his own head. That's a thing that probably would not happen in actual reality—very few real-world highly trained axemen accidentally decapitate themselves—, but that sort of spectacle is expected in reality-plus because Argath of Sartar is important, and his training took place during the game.
A game with fumbles tends not to mirror reality because the random number generator that's used has so little granularity. A highly trained archer might shoot himself in the head with an arrow or some other absurdity, but that surely won't happen once every 100 arrows he fires in actual reality! Were there even a 1% chance of every attempt leading to catastrophe in our reality, archery ranges would be sad, sad places, littered with the dead, naked, and injured.
But in reality-plus if those 100 arrows are fired while they matter—during the actual game—, one of those hundred arrows is likely to cause the archer's armor strap to break or make him fall and twist his ankle (those are average results on the Runequest (1980) Fumble Table). That happens in reality-plus because that's spectacular.
That possibility of spectacle makes the game, for some, more interesting rather than more ridiculous. It turns the game from what some may view as a mere simulation into a drama, albeit, in this fan's opinion, in a really forced way.
Best Answer
According to Wikipedia, this is called zero bias notation(retrieved Aug 31, 2010). You simply replace the “d” with a “z” and the old highest die value with the new one. So your example of three eight-sided dice marked 0–7 would be written as 3z7.
The only reference the Wikipedia article provided for this was an RPGnet post. I've never seen any signs that the notation caught on anywhere, but I do like it as a system.