[RPG] Can a Grappled Monk viably use Open Hand Technique to end the grapple

Yes, the monk can knock creatures of any size prone.

Other methods of proning specify their size limitations, for example, under Shoving a Creature:

The target of your shove must be no more than one size larger than you, and it must be within your reach.

And similarly, under the Battle Master's Trip Attack:

You add the superiority die to the attack's damage roll, and if the target is Large or smaller, it must make a Strength saving throw. On a failed save, you knock the target prone.

The Monk's Open Hand technique just says:

Whenever you hit a creature with one of the attacks granted by your Flurry of Blows, you can impose one of the following effects on the target:

• It must succeed on a Dexterity saving throw or be knocked prone.

Since it doesn't include any limitations or restrictions, there are none.

The next section is a textwall: The tl;dr is that prone just isn't that powerful.

As far as this being unbalanced is concerned, I think you're overestimating the impact of the prone condition. The effects of the prone condition are to make attacks from adjacent squares more accurate, and attacks from a distance less accurate, as well as making the prone creature's attacks less accurate.

None of this is a particularly big deal, and since any creature can end the prone condition just by spending half their movement, it's unlikely they'll ever feel the penalty imposed on their attacks. In effect, the prone condition merely increases the accuracy of attacks from adjacent squares until the creature's turn, then costs them half their movement.

Furthermore, for this feature to work, it requires a successful attack followed by a Dexterity saving throw. The target creature effectively gets 2 chances to defend against this effect. It's also worth considering that some monsters (for example, elementals) are immune to the prone condition, so this simply won't work on them at all.

Finally, remember that ki points are the Monk's main resource, and they are quite scarce, especially at low levels. Spending ki points is meant to have a significant impact, otherwise they wouldn't be quite so rare. If the effects you got from them weren't useful, the Monk class would just be a weaker version of the Fighter.

You can apply an effect to each hit you make on an enemy. The key phrasing here is:

Whenever you hit a creature with one of the attacks granted by your Flurry of Blows, you can impose one of the following effects on that target[...]

So whenever you land an attack, you can apply an effect. If you land multiple attacks, you can apply an effect on each hit. That said, there are a few problems with your idea. The first is that Flurry of Blows only grants two attacks:

Immediately after you take the Attack action on your turn, you can spend 1 ki point to make two unarmed strikes as a bonus action.

The second problem is that you don't make the attacks simultaneously. When you hit with the first attack, you roll damage and apply one of the effects. From page 194 of the PHB:

Whether you’re striking with a melee weapon, firing a weapon at range, or making an attack roll as part of a spell, an attack has a simple structure.

1. Choose a target. Pick a target within your attack’s range: a creature, an object, or a location.

2. Determine modifiers. The DM determines whether the target has cover and whether you have advantage or disadvantage against the target. In addition, spells, special abilities, and other effects can apply penalties or bonuses to your attack roll.

3. Resolve the attack. You make the attack roll. On a hit, you roll damage, unless the particular attack has rules that specify otherwise. Some attacks cause special effects in addition to or instead of damage. If there’s ever any question whether something you’re doing counts as an attack, the rule is simple: if you’re making an attack roll, you’re making an attack.

Each attack resolves separately, including the special effects you add to it. So you can't hit the enemy twice to push them back 30 feet.

What you can do is push the enemy back 15 feet, then run up to them and then hit them again, and push them back another 15 feet. The exact rule for moving between attacks is (PHB, page 190):

Moving between Attacks

If you take an action that includes more than one weapon attack, you can break up your movement even further by moving between those attacks. For example, a fighter who can make two attacks with the Extra Attack feature and who has a speed of 25 feet could move 10 feet, make an attack, move 15 feet, and then attack again.