Excellent question! I currently have a player who, literally, wishes he could be a cross between Wulfgar from the Forgotten Realms novels and Rowdy Roddy Piper. Putting good taste aside, he requested that we homebrew a hulking, grappling character in the vein of professional wrestlers. I'll share my thoughts on that homebrew and some of the 'moves' we have developed over time.
Caveat emptor: The list of moves is quite extensive, so my apologies. Additionally, my homebrew skills are crude compared to, say, KRyan, so your mileage may vary.
Class Basics
The vision of this class is of a flashy, muscled brute who wades (or dives!) into enemies and manhandles them, dealing low to moderate damage to (generally) single targets while granting allies advantage on those targets. To keep the character alive long enough to be enjoyable we looked to the Barbarian template (PHB: 46), to offer excellent hit dice (d12) and Unarmored Defense. To keep them effective as levels increased we took items from the Fighter template (PHB: 70) to offer Action Surge, Extra Attack (3), and Indomitable.
Hit Dice: 1d12 per Wrestler level.
Armor Proficiency: None
Weapon Proficiency: Simple
Saving Throws: Strength, Constitution
Skills: Choose three from Athletics, Acrobatics, Intimidation, Insight, Medicine, Performance, Perception, and Survival
Tools: Painter's Supplies, Disguise Kit, Leatherworker's Tools — Limited to costume creation.
Unique Resource: Flair
The Wrestler uses their knowledge of wrestling techniques and their penchant for flamboyance to power some abilities and attacks. Flair Points = Wrestler Level. Flair recharges after a Long Rest. After LVL 9, Flair recharges after a Short Rest.
Wrestler
You are a professional wrestler and natural showman. At 1st level you have access to Ungrappled Moves and Grappled Moves as detailed in the Moves section below. Flair is the resource that powers these abilities. You have Flair Points equal to your Wrestler level, as shown in the Flair Points column of the Wrestler table.
Unarmored Defense
While you are not wearing any armor, your Armor Class equals 10 + your DEX modifier + your CON modifier. You can use a shield and still gain this benefit.
Airborne!
Beginning at 2nd level you gain access to Aerial Moves as detailed in the Moves section below.
Combo
At 3rd level you may string up to two Moves together. You must pay the cost for each Move as listed in the Move table. The target may make a STR or DEX save (their choice) vs. your Wrestler DC between Moves to break the combo. If successful, the combo ends, they take no additional damage, and your grapple is broken.
Rolling Thunder
At 3rd level, as a bonus action, you may roll up to 10' in any direction. This movement allows you to move through an enemy's space and does not trigger Attacks of Opportunity.
Ability Score Improvement
When you reach 4th level, and again at 8th, 12th, 14th, 16th, and 19th level, you can increase one ability score of your choice by 2, or two ability scores of your choice by 1. As normal, you can't increase an ability score above 20 using this feature.
Extra Attack
Beginning at 5th level, you can attack twice, instead of once, whenever you take the Attack action on your turn. The number of attacks increases to three when you reach 13th level and four when you reach 18th level.
KO
At 5th level, if the last move of a Combo would reduce a grappled enemy to less than 20% HP you may attempt to KO that enemy. The creature must make a CON save vs. your Wrestler DC. If they fail, you perform an extra Move and reduce the target to 0 HP. This ability triggers at higher HP thresholds at 11th level(25%) and 17th level (35%).
Action Surge
Starting at 6th level, you can push yourself beyond your normal limits for a moment. On your turn, you can take one additional action on top of your regular action and a possible bonus action. Once you use this feature you must finish a short or long rest before you can use it again. Starting at 17th level you can use it twice before a rest, but only once on the same turn.
Remarkable Athlete
Starting at 7th level, you can add half your proficiency bonus (round up) to any Strength, Dexterity, or Constitution check you make that doesn't already use your proficiency bonus. In addition, when you make a running long jump, the distance you can cover increases by a number of feet equal to your STR modifier.
Knack for Flair
At 9th level you regain half of your missing Flair Points at the end of a short rest. You must spend at least 30 minutes focusing on your machismo(a), (working on poses, one-liners, taunts, etc) to regain your Flair Points.
Improved Grapple
At 9th level you have learned to grapple enemies with improved efficiency. You gain a +1 bonus on all grapple checks. Additionally, you may attempt to grapple two enemies at once. The enemies must be within 10' of each other and you must have adequate movement to reach them both. Each enemy may contest your grapple check.
Tandem Move
At 10th level you gain access to Tandem Moves as detailed below in the Moves section. Tandem Moves are denoted by Tandem in the Move description. Tandem Moves not specifically listed in the Tandem Moves section may be used vs. a single creature.
Improved Combo
At 10th level, you may string up to three moves together. You must pay the cost for each Move as listed in the Move table. The target may make a STR or DEX save (their choice) vs. your Wrestler DC between moves to break the combo. If successful, the combo ends, they take no additional damage, and your grapple is broken. Additionally, the first move in a combo costs no Flair Points.
Indomitable
Beginning at 11th level, you can reroll a saving throw that you fail. If you do so, you must use the new roll and you can't use this feature again until you finish a long rest. You can use this feature twice between long rests when you reach 15th level.
Eye of the Tiger
Beginning at 13th level, you can harness your machismo(a) to intimidate your enemies. When you do so, you gain Advantage on all Intimidation checks until your next turn. You may use this ability a number of times equal to your STR modifier (minimum 1) between long rests.
Combo Master
Starting at 15th level you may string up to four moves together. You must pay the cost for each Move as listed in the Move table. The target may make a STR or DEX save (their choice) vs. your Wrestler DC between moves to break the combo. If successful, the combo ends, they take no additional damage, and your grapple is broken. Additionally, the first move in a combo costs no Flair Points and if all four moves are successfully completed you are refunded one Flair Point.
Grappling Master
At 18th level you have mastered the art of grappled combat. You gain a +2 bonus on all grapple checks and have advantage on all STR and DEX checks related to Moves and grappling in general.
Commensurate Showman
At 20th level, whenever you roll initiative and have 0 Flair Points you regain 4 Flair Points.
Moves
Moves are the core of this homebrew class. Using attacks to tackle, grapple, pin, and otherwise physically abuse single, and occasionally multiple, targets, the Wrestler acts as a disruptive force in the midst of the enemy line. Their moves push, pull, throw, grapple, and pin enemies, neutralizing their threat to other party members and allowing advantage on party members' attacks.
Moves that cost Flair Points require 1 Action to complete unless otherwise noted.
Ungrappled Moves
Bell Ringer: You clap your hands on opponent's ears, potentially stunning them.
1 Flair Point. 1d4 + STR mod + CON save or stunned until the next turn.
Crossbody Block: You leap sideways at an enemy and attempt to bear them to the ground.
1d4 + Grappled + Prone (both)
Armbar: You grab an enemy's arm and twist, putting pressure on the elbow and shoulder
The enemy is Grappled.
Scissor Press: You leap leg-first at an enemy, bearing them to the ground in a scissor hold.
1 Flair Point. 1d4 + Grappled + Prone (enemy). Next round, if the enemy does not succeed on their STR check to break the grapple, the Wrestler may use one attack to deal 1d6 + STR constriction damage.
Clothesline: If you have moved 10′ toward an enemy this round you may attempt to clothesline them. If successful, you move an additional 5′, through the enemy's space.
1d6 + Prone + CON save or stunned.
At higher level: For each Flair point you use you may attempt to apply this effect to an additional target. You may use both arms, affecting two creatures standing adjacent to each other. You must have sufficient movement to continue your charge through the last creature affected.
Sledgehammer: You lock your fists together and deliver a massive blow the your enemy.
1d6 + STR mod. If two or more consecutive blows are landed the enemy must make a DEX save vs. your Wrestler DC or be pushed back 5′ for each blow after the 1st.
Helicopter: You whirl in place, roaring with fists extended, bludgeoning all those around you.
1d4 + STR in a 5′ sphere centered on you.
At higher level: For each Flair Point you expend, proc the ability one additional time.
Elbow Drop: You leap into the air, driving your elbow into a prone enemy.
1d6 + STR. For each 5′ you drop, add 1d8 to enemy damage and take 1d4 damage. Normal falling damage is halved when using this move.
Grappled Moves
These moves become available once an enemy is grappled, or combo into moves that require the enemy to be grappled.
Hammerlock: You grab an enemy's arm and pull it up behind their back, securing your grip on their neck.
1 Flair Point. Opponent is Grappled and has Disadvantage on attacks.
Bear Hug: You grab your enemy and clutch them tightly to your chest, lifting them off the ground.
1 Action. 1d6 + STR and enemy is grappled. If they fail to break the grapple next turn you may apply 1d8 + STR constriction damage.
Combo: You may spend a Flair Point to combo a Bear Hug with a Suplex for an additional 1d8 + STR damage. The enemy is prone.
Reverse Grapple: You move back-to-back and lock your arms under theirs, lifting them from their feet and bending them over your back.
Flair Point. 1d4 + STR and the enemy is grappled. If they are humanoid they cannot make weapon attacks or cast spells with a somatic component.
Combo: You may spend an additional Flair Point to combo a Reverse Grapple with a Facebuster for an additional 1d8 + STR damage. The enemy is prone and remains grappled.
Half Nelson: You pass your arm under the arm of a prone enemy, gripping their neck.
1d4 + STR and the enemy is grappled.
Combo: You may spend a Flair Point to combo a Half Nelson with a Nelson Choke (kata ha jime), doing 1d8 + STR damage. If the enemy does not break the hold you may deal this damage again at the start of each of your turns.
Full Nelson: You pass both your arms under the arms of a prone enemy, gripping their neck and pinning them to the ground.
1d6 + STR and the enemy is grappled. The may not make weapon attacks and may not cast spells that require a somatic component.
Piledriver: You grapple your enemy, turn them upside-down, and drop to a sitting position, driving their head into the ground.
1 Flair Point. 2d6 + STR damage and the enemy is grappled.
Combo: You may expend an additional Flair Point to execute a Scissor Choke, wrapping your legs around your opponent's face and neck and squeezing for an additional 1d6 + STR damage. The grappled opponent may not cast spells or use abilities that require a verbal component. If they fail to break the grapple next turn they take an additional 1d6 damage and begin to suffocate.
Aerial Techniques
These techniques involve the Wrestler becoming airborne and/or causing their enemies to become so.
Windmill: You grab a prone enemy and hoist them into the air, spinning them around and using them as a weapon.
1 Flair Point. 1d4 in a 5′ sphere centered on you. Hitting a solid object, such as a wall, halts the spin. The spinning enemy takes 1d4 damage for each other creature and object they impact.
At higher level: For each Flair Point you expend you may move 5′ in any direction, up to your total movement, proccing this ability once for each 5′ you move.
Combo: Toss. At the end of the spin you may make a STR check vs DC 15. On a success you may throw the grappled opponent up to 15′ in any direction. They take 2d4 damage from the impact.
Double Kick: If you have moved 10′ toward an enemy you may perform a double kick, attempting to plant both your feet in their chest and bear them to the ground.
1 Action. 2d4 + STR. The enemy must make a DEX save or be knocked back 5′ and prone.
Summary
This homebrew Wrestler is clearly still in the iterative phase. We have taken the character through 7th level and no abilities appear frankly broken. Flair as a resource appears well-balanced, but this player is not a min/maxer so I cannot say things can't be abused. Damage was scaled back due to the inherent advantage granted to allies when enemies are grappled. No level table has been created yet — we are essentially making this up as we go along.
The impact the class has had on the game has been generally positive. I find that the class provides ample opportunities to apply the Rule of Cool. My wrestler-obsessed player very much enjoys the build and the other players at the table enjoy the antics and do not find that they detract from their own fun.
Abilities were sourced from Wikipedia's “Professional wrestling attacks” article. We have not progressed past two-move combos, but it would be a small thing to string suitable moves together (at a cost, of course). Your player likely has a good vision as to the types of things he would like to do. Take those into account and build new moves similar to the ones above or re-skin existing moves to fit the flavor you are seeking.
We are considering using crafting to allow the creation of different costumes which may provide minor benefits such as +2 to intimidation checks (face paint!) or +1′ or +2′ jumping distance (capes make you fly farther).
I hope you and your players find this to be both helpful and enjoyable.
Best Answer
No, there is no official "Adventure Zone D&D Ruleset"
They haven't described the changes they've made to the rules, because that would be boring to most people.
I have listened to every episode, some multiple times (it's my favorite podcast). Griffin has never mentioned a specific set of rules, nor are they listed anywhere on their Maximum Fun page or their Tumblr.
The reason for this is that they want the show to be entertaining for people who don't play RPG's, it's a comedy podcast first, a D&D podcast second. They don't want to be bound to any rules, even of their own making, because don't want rules to get in the way of a fun show.
There are a few rules that they seem to have decided beforehand to ignore, like XP (they generally level up their characters after they complete a story arc) and inventory management (they don't worry about encumbrance or buying uninteresting things like trail rations, bedrolls, etc.) Keeping track of these sorts of things would be very boring for most people. Also, Griffin has made up most of their magic items to fit the setting and to be interesting.
Other than that, Griffin makes rule decisions on the fly based on what would make the most entertaining show. They rarely refer to the rule book (except when Merl or Taako cast a spell they've never used before) because that would be boring for most people.
Keep in mind a significant portion of the listeners of the show don't care about rules, they listen to the show because they are fans of the McElroy brothers.
That being said, apart from the things I listed above, I haven't noticed many significant changes to the rules. I'm sure someone more familiar with 5e could point out a few more differences, but as far as I can tell the game they play isn't hugely different from RAW.
Update: In the episode The The Adventure Zone Zone the cast discusses the show and answers some listener questions, much of their discussion is relevant to this topic. They do not spell out a set of rules, but they talk about how they decide what rules to follow and why they do it that way.