Say I'm a Battle Master using a 2-handed weapon, and I have Riposte, Trip Attack, and the Sentinel feat, and there's a friendly rogue beside me.
If a 5-feet creature hit me, and misses, I can Riposte, then apply the rule 1 of Sentinel to make its movement 0 for that round. Then when my turn comes I can use Trip Attack to make it go prone. Does this mean that when its turn comes, the enemy can't stand up because it has no movement?
Also, if it hits the rogue, could I use Sentinel rule 3 and then apply Trip Attack to make the creature prone and have 0 speed? Is that legit/viable?
No. No. Mostly no.
One at a time:
Can I apply Sentinel rule 1 to a Battlemaster's Riposte?
No. You can't because the Riposte isn't an opportunity attack. Sentinel rule 1 only works on opportunity attacks.
Your confusion on this point is understandable. Ripostes and Opportunity attacks are both (usually) melee weapon attacks that use your reaction. But that doesn't mean that every reaction attack is an opportunity attack.
Sentinel strict applies to opportunity attacks, not Ripostes, Ready Actions, or other attacks that you might use your reaction for.
Opportunity attacks only occur if a creature leaves your reach (or another says otherwise).
If I debuff a creature's speed with Sentinel and trip it, will it be stuck on its next round?
No. The movement debuff lasts for the rest of the turn, not the round (see quote above). A turn is one creature's chance to act. So if a monster moves out of reach during its turn, your opportunity attack also takes place during its turn. As soon as the next creature starts acting, its speed will no longer be 0.
Could I use Sentinel rule 3 and then apply Trip Attack to make the creature prone and have 0 speed?
Mostly no. Again, the attack that Sentinel provides you is not an opportunity attack.
If instead of hitting the rogue, the monster moved out of your reach, then it would be an opportunity attack, and you would be able to trip and debuff their speed. So make sure the rogue knows not to stand right next to you.
Bonus question: Can I use Trip Attack with Sentinel rule 3?
Yes. The only requirement for Trip attack is that you hit with a weapon attack (it doesn't even need to be melee). You can use this with the Attack action, bonus action attacks like Two-weapon fighting, Opportunity Attacks, Sentinel rule 3 attacks or pretty much anything else as long as its a weapon attack.
Different maneuvers have different requirements, so be sure to read each one carefully (for example, sweeping attack requires a melee weapon attack). And remember, only 1 maneuver per attack.