The penalty is being in difficult terrain

You're in difficult terrain for all creatures occupying the same space:

The space of another creature, whether hostile or not, also counts as difficult terrain.

The constraint is that creatures in the situation must attempt to move out of it.

The rules have a provision that generally avoids the stacking situation:

Whether a creature is a friend or an enemy, you can't willingly end your move in its space.

Grappling an unwilling ally will arrest them in your square: a clever edge case.

You can grapple your ally, providing they're unwilling, as grapple is a contest[1,2]. Sometimes one has to hold back their friend. The no-contest version is not a grapple, but is covered under the drag/lift/carry rules.

Cleverly grappling your ally to arrest their motion will work for a round. On the next round, both the grappler and the grappled characters must to attempt to move out of the square as long as they're both occupying it. Not attempting to move out of the square would mean they're willingly ending their move there, and that's not permitted per the rule cited above.

This is an example of the edge case of the rules and can be further manipulated into a broken situation that ends with a DM narrating when it stops.

The rules do not state a limit of the number of creatures in a space, because 5e rules are not a physics engine.

There are currently not specific rules about the number of creatures that can occupy a space. The provision is that they can't willingly do so in combat.

Making a large stack of creatures is similar to such endeavors as a peasant railgun or a hired army running through a teleportation circle.

A creature usually can't take a 5-foot step to enter a square that hampers the creature's movement

When determining the movement costs for hampered movement, the game usually doesn't care where the creature is and instead cares where the creature's going:

Difficult terrain, obstacles, and poor visibility can hamper movement (see Table: Hampered Movement for details). When movement is hampered, each square moved into usually counts as two squares, effectively reducing the distance that a character can cover in a move.

If more than one hampering condition applies, multiply all additional costs that apply. This is a specific exception to the normal rule for doubling.

In some situations, your movement may be so hampered that you don't have sufficient speed even to move 5 feet (1 square). In such a case, you may use a full-round action to move 5 feet (1 square) in any direction, even diagonally. Even though this looks like a 5-foot step, it's not, and thus it provokes attacks of opportunity normally. (You can't take advantage of this rule to move through impassable terrain or to move when all movement is prohibited to you.)

You can't run or charge through any square that would hamper your movement.

Emphasis—and extra emphasis—mine. Thus at issue is the creature's destination square not its current square. The creature's already in its current square, so the creature doesn't have to care about that square's terrain, having already overcome the difficulty of entering it!

For example, going from a square of supernatural darkness into an adjacent square of bright light is just normal movement, but going from a square of bright light into an adjacent square of supernatural darkness usually costs double movement due to poor visibility. In fact, going from any square into an adjacent square of supernatural darkness usually costs double movement due to poor visibility, the condition of the starting square typically not mattering at all.

Exiting then reentering the same square of hampering terrain still costs additional movement, of course. A creature can't move normally back and forth between squares of hampering terrain just because it's occupied the same square of hampering terrain previously. For example, moving from a square of supernatural darkness to a square of supernatural darkness then back to the original square of supernatural darkness does not speed up the return trip.