[RPG] Is it an action to voluntarily end a grapple

Your players have a lot of options.

From the Grappled condition:

• The condition ends if the grappler is incapacitated (see the condition).
• The condition also ends if an effect removes the grappled creature from the reach of the grappler or grappling effect, such as when a creature is hurled away by the thunderwave spell.

So any method of inflicting the incapacitated condition on the mind flayer will end the grapple. Likewise, anything that gets the barbarian out of the mind flayer's reach will also end the grapple. There are far too many ways of achieving these to list here.

It's also worth mentioning that, since the Extract Brain ability requires the mind flayer to use an action, preventing the mind flayer from taking actions would also work. Most of the methods to do this would involve inflicting the incapacitated condition, and therefore end the grapple anyway. But as an alternative option, there's always mind control. There are a large number of spells that could be used to persuade the mind flayer not to extract the barbarian's brain, or just to let them go.

Since the mind flayer's grapple requires a Medium or smaller creature, polymorphing the barbarian into something Large should also free him. There is a minor point of ambiguity here - while the grapple can only be used on a Medium or smaller creature, it doesn't technically say anything about what happens if a grappled target becomes Large. But I think most DMs would agree that this should work.

While it's not an exact analogue to 3.5e grappling, I believe the closest thing to what you're looking for is the Grappler feat. It gives you advantage on attack rolls against any creature you have grappled. It also allows you to give your grappled opponent the Restrained condition as well as the Grappled condition when you grapple them. This inflicts the following penalties:

• A restrained creature’s speed becomes 0, and it can’t benefit from any bonus to its speed.

• Attack rolls against the creature have advantage, and the creature’s attack rolls have disadvantage.

• The creature has disadvantage on Dexterity saving throws.

Note that when you give your grappled opponent the Restrained condition, you also give it to yourself.

I should point out that, just like in 3.5e, if you want to be good at grappling, the best way to do it is just to be a Druid and turn into something that is inherently good at grappling. Many monsters automatically grapple and restrain on attack, without restraining themselves.

Good, early available forms for this include the Giant Constrictor Snake, the Giant Octopus, and the Giant Toad. Later on, the Giant Crocodile or the Water Elemental is probably your best bet. Using polymorph, the T-Rex is particularly nice, but it's not ideal to use polymorph on yourself, because of concentration.