A creature that, because of the creature's size, provokes attacks of opportunity for entering a foe's square provokes attacks of opportunity when entering that foe's square even if the creature enters that square by taking a 5-ft. step...

The rules cited in the question are from two different places about two different things. The 5-ft. step rules don't care about the creature size rules, and the creature size rules don't care about the 5-ft. step rules. The two rules are managed independently and don't interact. Thus, when a Tiny, Diminutive, or Fine creature enters an opponent's square, it provokes an attack of opportunity even if it entered that opponent's square by taking a 5-ft. step. That's a function of the creature's size exclusive of the creature's 5-ft. step.

Example 1
A Tiny, Diminutive, or Fine creature takes a 5-ft. step when adjacent to opponent and within that opponent's threatened area that doesn't provoke an attack of opportunity (because of the 5 ft. step) and moves into his opponent's square which does provoke attacks of opportunity (because of the creature's size).

When a Tiny, Diminutive, or Fine creature takes a move action to move within its opponent's threatened area then enter its opponent's space, the creature probably should provoke attacks of opportunity twice, once for the movement within the creature's threatened area and once for entering the creature's square, except that the FAQ (see below) says to treat the opportunity granted by moving within a creature's threatened area as part of the same movement needed to enter a creature's space. Thus a typical Medium creature gains only one attack of opportunity against a Tiny foe that takes a move action to enter the creature's square.

Example 2
A Tiny, Diminutive, or Fine creature takes a move action to travel 30 ft. to enter an opponent's square. The creature provokes an attack of opportunity when moving within the opponent's threatened area as normal then can enter the opponent's square with impunity.

It's sort unfortunate that two attacks of opportunity aren't provoked. It might've saved the Dexterity 13 commoner with the feat Combat Reflexes from death by house cat.

...And the FAQ agrees

The Pathfinder Roleplaying Game Core Rulebook FAQ contains the following exchange:

Question: In the section on Tiny and smaller creatures, it says that entering a creature’s space provokes an attack of opportunity, but typically 5-foot steps don’t provoke an attack of opportunity. If a Tiny or smaller creature took a 5-foot step into a creature’s space, would it provoke an attack of opportunity?
Answer: Yes. Even with a 5-foot step, a Tiny or smaller creature entering a creature’s space provokes an attack of opportunity (unless it is using a more specific ability to avoid the attack of opportunity such as the Monkey Shine feat). This doesn’t mean that a Tiny or smaller creature entering a creature’s space and moving out of a threatened square with a move action provokes two attacks of opportunity from that creature, for the same reason that moving out of multiple of a creature’s threatened squares in the same move action doesn’t provoke two attacks of opportunity.

This exchange was added to the FAQ in Jan. 2016.

The penalty is being in difficult terrain

You're in difficult terrain for all creatures occupying the same space:

The space of another creature, whether hostile or not, also counts as difficult terrain.

The constraint is that creatures in the situation must attempt to move out of it.

The rules have a provision that generally avoids the stacking situation:

Whether a creature is a friend or an enemy, you can't willingly end your move in its space.

Grappling an unwilling ally will arrest them in your square: a clever edge case.

You can grapple your ally, providing they're unwilling, as grapple is a contest[1,2]. Sometimes one has to hold back their friend. The no-contest version is not a grapple, but is covered under the drag/lift/carry rules.

Cleverly grappling your ally to arrest their motion will work for a round. On the next round, both the grappler and the grappled characters must to attempt to move out of the square as long as they're both occupying it. Not attempting to move out of the square would mean they're willingly ending their move there, and that's not permitted per the rule cited above.

This is an example of the edge case of the rules and can be further manipulated into a broken situation that ends with a DM narrating when it stops.

The rules do not state a limit of the number of creatures in a space, because 5e rules are not a physics engine.

There are currently not specific rules about the number of creatures that can occupy a space. The provision is that they can't willingly do so in combat.

Making a large stack of creatures is similar to such endeavors as a peasant railgun or a hired army running through a teleportation circle.