[RPG] How to players work together to take actions that are otherwise impossible

This is how skills are supposed to work!

If you are in a situation where there is only one person doing something, and they are rolling a single skill check, then yes, this is how it's supposed to work. Giving help is a natural thing and should be used in situations like this. There is no reason to prevent it unless the task is clearly something that's not going to benefit from someone else giving you assistance. There are some things you can do to limit it.

It's also worth noting that helping can often save you some table time. As a AceCalhoun points out in the comments, in many cases what happens if you don't help is that everyone in the party tries their hand at the task. This behaves very much like advantage, but with a slightly lower overall modifier (because most likely you'll have one character who is good at a task and the rest that are lower). So Working together only raises the change of success slightly and consumes less table time in these cases.

• Be a bit more stringent about what you allow for assistance. Is coaching stealth really all that helpful? do you really want someone looking/talking over your shoulder while you're picking that lock? Evaluate situations where characters attempt to aid more carefully.

• Have more than one thing going on at once. If all the characters need to be stealthy, they can't be helping each other. And if you need two arcane characters working on the sigils on opposite sides of the room, maybe they have to choose which one gets help from the third (or don't have anyone to help at all).

• Make things take multiple rolls and limit helping on all of them. Maybe the first roll the wizard can be helped, but after that he's on the other side of the trap, or arm deep in the sigil or something to where additional assistance isn't going to help him.

• Figure out how to inflict disadvantage for the task. Maybe there are mitigating circumstances.

• Create a distraction. A rogue can't help the wizard if he's busy fighting baddies. Make some skill checks happen in an occupied room. Make completing the skill checks the win condition rather than defeating the enemies.

The basic crux of all of this is that helping is supposed to a mundane task that provides advantage. Yes, that's a huge deal, but it also doesn't stack with other things that give you advantage and it can be easily cancelled by disadvantage.

So get creative! Build some situations into your adventures that prevent your heroes from helping each other (or make the opportunity cost higher). But don't do it all the time, that might get tiresome. Adventurers like to help each other out, let them, but don't make it easy all the time.

If the odds are that stacked in favor of either side, don't call for a roll

The Basic Rules pg. 58 and Player's Handbook both describe ability checks with the following prelude (emphasis mine):

The DM calls for an ability check when a character or monster attempts an action (other than an attack) that has a chance of failure. When the outcome is uncertain, the dice determine the results.

When cumulative advantages stack that heavily against the player characters, the outcome is not uncertain; the player characters shouldn't succeed. So instead of letting them roll, just tell them that the task is impossible.

Similarly, if they have so many advantages stacked up on their side that it seems ridiculous that the PCs have a chance to fail, don't have them roll at all; just declare that they succeed. So in the case of your DC 30 portcullis trap, if you feel that the combined strength of the player characters is enough to lift the portcullis, then don't roll and instead declare that they succeed in lifting the thing.

In this way, you are only rolling for contests which are possible at all and don't have to worry about situations in which either PC success or failure is almost impossible to justify.

But what if their combined strength is only enough to justify a chance at success?

You may end up in situations, like the trap you describe, where the combined effort of every PC is enough to have a chance to succeed, but is not enough so that success is guaranteed. In this case, there aren't really many rules to help you. 5e is designed with simplicity at heart, so there are no rules for circumstantial bonuses. With that said, you are the DM and you have control.

In the same ability check section, the basic rules and PHB say:

For every ability check, the DM decides which of the six abilities is relevant to the task at hand and the difficulty of the task, represented by a Difficulty Class.

You are in control of the DC, and you get to decide what it should be. You can even decide to change it on the fly to make your game work for you. We essentially solve the problem in reverse. Instead of looking for rules to modify the bonus to the roll, we leverage the power the rules give us as a DM to modify the target number.

You have every right to change the DC to whatever value you feel is most appropriate.

For example, 5 people lifting a portcullis might not be impossible, it might just be hard. So you can adjust the DC down from 30 to ~20 (based on the chart on typical difficulty classes in the basic rules and PHB).

You can even choose values between these markers. So, the task might be more than medium but not quite hard. In that case, you can decide "five people makes this a DC 17 or 18, not a 30". And then you can use either one roll from one of the players (with or without advantage at your discretion) or the group check rules now that the goal is much easier for everyone in the party to make.

This is the approach I use the most often in situations like these and I firmly believe that it's much less of a headache than trying to deal with rules for circumstantial bonuses.