Mechanically, they are very different games.
- D&D 5th edition uses dice. Gloomhaven uses a deck of twenty cards which modify damage. You do not roll to hit; you automatically hit (except on a critical miss), and draw to see whether you deal more or less damage than average.
- D&D encourages players to work as a team. Gloomhaven encourages players to grab the treasure for themselves where possible - a faux pas in D&D - and forbids players from sharing treasure with characters who got less loot.
- D&D encourages players to develop their character and role-play (act in-character). This confers no benefit to Gloomhaven, which is purely a tactical miniatures game.
- Gloomhaven characters who "die" are merely exhausted, and come back to life at the end of a dungeon.
- D&D has a Dungeon Master who adjudicates rules and judges circumstances on the fly. Gloomhaven has no such role; all monster actions and events are decided by fixed rules.
- Gloomhaven's characters, adventures and world are fixed. D&D allows DMs to invent their own adventures, and allows players to go on adventures the DM didn't prepare, or invent new characters.
- Gloomhaven focuses primarily on tactical miniatures combat. It is, in a way, the tactical miniatures board game that D&D 4e's critics feared. D&D 4e also held this focus, but D&D 5e make miniatures purely optional, and allows players to focus on other aspects of the game.
Thematically, however, it's quite similar to D&D, specifically 4th edition. There is an elemental plane with mixtures of all elements, as the Elemental Chaos; there are demons who hail from this elemental realm. There are short and long rests. It implements D&D 5e's advantage system. The primary difference is that the player character races are largely original, making it difficult for D&D players to get a sense of each character's role before playing them.
To address some of your particular concerns:
- Knowing D&D won't help you select a Gloomhaven character. Gloomhaven's character races and classes are totally different to D&D, meaning you can't pick a familiar archetype before you play. Even an experienced D&D player is going in blind when they pick a Gloomhaven character.
- There is no character customization in Gloomhaven, beyond selecting your character's name, selecting from a small number of optional power cards, and changing cards at level-up.
- You cannot muddle through without knowing the rules correctly like in D&D. There is a lack of a GM to smooth things over.
Short answer: No, mostly.
I come from a D&D 3.5 background, so I've had ample experience with the extreme power difference that you can get between a straight-class Monk and an optimized Ruby Knight Vindicator. D&D 3.5 had a lot of complexity: Bloodlines, Templates, Prestige Classes, Organizations could all be mixed and matched, and some combinations were insane, really rewarding deep knowledge of the game.
By contrast, D&D 5.0 has far fewer options. None of the aforementioned Bloodlines, Templates, Prestige Classes, for starters. Simplified skills system. Fewer feats to choose from, etc...
The lack of flexibility makes for a much more balanced game, because designers were able to pay attention to a lot of interactions:
- Any single-class character with its Ability scores correctly assigned is viable.
- The power difference between two different "specialization" of a given class are not that big.
If we look at the example of the Two-Weapon Fighting Fighter, it is indeed less powerful than a Great Weapon Fighting Fighter.
There are two things to notice, though:
- It is not THAT much less powerful. We are not talking about a factor x2 here; taking a Str of 20 (+5) and 4 attacks, the formula given yields 47.5 DPR for TWF and 53.2 DPR for GWF. That's a mere ~12% difference.
- It allows using Dex rather than Str as your attack ability, which is actually good for optimization as Dex also gives Initiative, AC and saving throws whereas Str doesn't give much.
The worst part is equipment -- having to get two magic weapons -- as a DM that's really the easiest part to solve.
As you can see if you peruse the comments to this answer, there are apparently a few "traps" to avoid, such as Ranger (Beast Master) and Monk (Four Elements).
In general, though, it could happen that a player starts a character with a certain picture in mind, and it turns out in play that the character just does not live up to their expectations. Maybe they misread the rules, maybe they can hardly ever pull it off. It's happened to us all, really.
There are also choices that were superb at 1st level, and whose utility decrease as time passes.
In this case, you should allow the player to tweak their character's past choices. There are already classes which allow switching some choices at pre-determined intervals -- the Warlock switching known invocations, the Battle Master switching maneuvers -- so it's fairly easy to allow switching Fighting Style or Feat when gaining a level, and much less work than creating and integrating a new character in the campaign.
This approach alleviates the pressure from your players: if they make a bad choice, they are allowed to correct it; no sweat, no fuss, the show just goes on!
Unless your table is very concerned with optimizing, and it doesn't look so, then D&D 5.0 should really be a straight improvement.
I personally find it extremely freeing; in D&D 5.0 you can play a sub-par character because you had a cool idea you want to toy with, and yet still contribute to the party without requiring extra work for the DM because you are not that far below the power curve.
One class, one specialty, and you've got a viable character. It's that simple.
Best Answer
At the risk of downvotes, I'm not going to do a specific build, but present some options for you to consider.
Before beginning, I want to restate that a Grappler build is more of a support style because of the existing rules around the Grappled Condition. Adding feats to help support it can give you Advantage on attacks (Grappler Feat), but unless you want to make yourself Restrained along with your target (which takes you out of the combat as well), the most you can do is set a creature's speed to 0.
ABILITY SCORES
STRENGTH and CONSTITUTION will be your friends here. Grappling is a Strength-based check, so having the highest modifier is going to go a long way. When grappling, you are just as likely as to get hit as hitting the target, so a having HP to absorb that damage and keep the creature grappled is very important.
RACES
There are some races that will vastly improve your ability to Grapple and possibly even reposition enemies on the battlefield by increasing your ability to move them.
Goliath - +2 STR, +1 CON And you count as one size larger for determining carrying capacity and when you push/drag/lift. This can be a big deal for moving bigger creatures. *Talk with your DM if this means if you (a medium creature) can now grapple a Huge creature (normally you are limited to one size larger, PHB 195.)
Aarakocra - +2 DEX, +1 WIS. The scores aren't the big improvement here for you with this race. IF your DM allows for it, what you get from this race is natural Flight. Being able to take a creature up in the air with you and dropping them is pretty amazing.
CLASSES
There are a lot of options, each of which have different effects on how you can play your character.
Barbarian - Unarmored Defense could be big here (and would pair nicely with a MAD build of STR/DEX/CON on an Aarakocra.) When Raging you also get advantage on STR checks and saving throws which will help with your grapple. The danger with Rage is that if you don't take damage, you can lose it. Taking the Totem Warrior-Bear path gives you some additional capabilities - namely at 6th level you can double your carry capacity again.
Bard - Your main focus on this is to become a Lore Bard for their 3rd level ability Cutting Words. Unfortunately, the amount of times you can do this is limited by your Charisma modifier, which introduces another stat for you to have at a decent level. Cutting Words allows for you to reduce the ability check roll of a creature trying to break your grapple - a very big deal! By going Bard, you also get access to spells that can further bolster your grappling such as Enhance Ability as well as a multitude of other fun stuff. With bard, you also gain expertise in multiple skills which could give you additional utility outside of combat.
Fighter - Additional Actions! Battlemaster maneuvers to give you additional things to do when grappling! This is your most traditional that will net you a lot of capabilities with Medium/Large creature. The Action Surge gives you another chance to either attack for damage or make an another attempt to Grapple. Battle Master gives you some maneuvers, but most are based on weapon attacks, which you probably aren't doing with your Luchador concept. The Champion Fighter may be a better choice to improve your criticals (although your unarmed damage die isn't big.)
BACKGROUNDS
When looking here, finding something that might give you Athletics proficiency is what you're going for (if your class doesn't.)
Outlander - Athletics and survival skill proficiencies.
Sailor - Athletics and Perception skill proficiencies.
Soldier - Athletics and Intimidation skill proficiencies.
FEATS and ASIs
Getting your Strength score high is your top priority, but there are feats that can also increase your grappling:
Grappler - as has been said, this feat is problematic. Getting advantage on attacks is good, but if you can force the opponent prone while grappled, that is going to do more for you and your team. The other ability of giving the Restrained condition is powerful, but it also takes you out of the fight.
Tavern Brawler - Proficiency with unarmed strikes works well with a Luchador concept. You also can still attack and have an opportunity to use a bonus action to grapple. The biggets downside is that your unarmed strikes are just d4+STR modifier. Not a big damage die.
Sentinel - Since grappled targets can still attack, having a feat that gives you an opportunity to use your reaction to hit them again is a nice
Lucky - Focusing on Grappling relationship to Lucky, this let's you reroll a bad ability check roll you have made.
HOMEBREW FEATS
If your DM allows pulling feats from this list, then you've got some new possibilities.
Quick Grappler - Bonus action to restrain rather than full action (can sideline the two of you in one round)
Brawler - d6 unarmed damage die
Expert Grappler - Can get cover from the grappled creature and deal STR modifier in damage 1/round.
MULTICLASSING
I would like to reference the answer andras gave on this one. I think it's an excellent suggestion if you want to multiclass rather than stick with a single class.
PARTY SUPPORT
Another thing to consider is if other party members can help support you. Anyone with Enlarge/Reduce can help you grapple the really big things.