[RPG] Does Minor Illusion break only when someone spends an action to investigate it

Depends on the situation.

There are two things to consider.

The target rationalizes any illogical outcomes from interacting with the phantasm. For example. a target attempting to walk across a phantasmal bridge that spans a chasm falls once it steps onto the bridge.

and

The target can use its action to examine the phantasm with an Intelligence (Investigation) check against your spell save DC

Given that the targeted creature failed the first save**, no matter how smart a creature is, if it rationalize any illogical outcome it would not use any action to investigate the illusion. This hold true for any direct interaction between the illusion and the target.

But, if there is a situation where the affected creature has a reason to believe it might be an illusion, it can use its action as the rule states. For example, an ally seeing him fighting with a ghost might help him yelling "it is an illusion" or "it is in your head". In this case, the creature has a reason to suspect is an illusion an use its action accordingly.

** The description of phantasmal force says:

The target must make an Intelligence saving throw. On a failed save, you create a phantasmal object, creature or other visible phenomenon of your choice that is no larger than a 10-foot cube and that is perceivable only to the target for the duration. This spell has no effect on undead or constructs

Well done!

Congratulations to your player in coming up with an imaginative use for the spell. Congratulations to you for enabling it and working out simple, practical mechanics for it.

I personally would have made it a Dexterity (Acrobatics) ability check rather than a saving throw as it is an attempt to keep your balance rather then actually resist the spell and also because more creatures have proficiency in Acrobatics than Dexterity saving throws. I would also consider Strength (Athletics) to stand your ground and give the target the choice.

I would also rule that creatures with other than 2 legs get advantage due to their greater stability.

Is this OP?

Does it break the rule about causing damage? Clearly not.

However, what I think you are asking is: is this overpowered for a cantrip?

My take is probably not and here's why:

• The wizard is giving up their action to possibly knock a single enemy prone. This means they are not using that action for something else like casting a higher level spell or a different cantrip. The chance is 70% for a modifier of 0 on the roll (going down by 5% for each +1).
• If the wizard instead used their action to Help they would guarantee advantage for one of their companions.
• "Fool me once shame on you, fool me twice shame on me". Guess who the creature is targeting next round, you know, while the wizard is concentrating on the Mold Earth cantrip? With a ready action, the caster is concentrating on the spell from their turn up to the trigger point.
• Anyone with a decent Strength (Athletics) modifier can do this as one of their attacks in an Attack action. If they have the same modifier they are better than the wizard (77.25%) and they are even better as the opponent's modifier increases. If they have multiple attacks, they can do this and still attack themselves.
• Don't even get me started on someone with expertise in Strength (Athletics) and the Shield Master feat. Grapple, knock prone with a bonus and they can't get up because their movement is 0.

Deeper concerns

Readying action requires you to "decide what perceivable circumstance will trigger your reaction." That is perceivable to the character. "When it completes its turn" is perceivable to the player, not the character.

I'm not saying it can't be done, I'm just saying it can't be done with that trigger.

"When it attacks, moves or casts a spell" is a workable trigger but there is the possibility that it does none of those things and the wizard is left with nothing to hang their action on.