Your PC can invent a new technique on his own

Taken from wikipedia's entry on the Southern Praying Mantis kung fu style:

Praying Mantis

The association of the term "Praying Mantis" with the style is also controversial. Each branch of the style offers a different explanation.

The traditions of the Chow Gar and Kwong Sai Jook Lum branches each maintain that their respective founders Chow Ah-Nam and Som Dot created their styles after witnessing a praying mantis fight and defeat a bird. Such inspiration is a recurring motif in the Chinese martial arts and can be found in the legends of Northern Praying Mantis, both White Crane styles, T'ai chi ch'uan, and Wing Chun.

If your rogue is observant enough - he may become inspired by some natural phenomenon, gaining insights about efficient and effective motions useful for offense, defense and maneuvering.

He doesn't need to be an acetic philosopher (though he may grow to become one as he advances) - he just needs the conviction and self-discipline to push him to go through the physical training required to hone his technique.

If your group's play style allows placing more emphasis on narrative at the cost of poorer combat optimization, you can even select to make the transition of your PC's reliance on his new martial skills gradual - continue wearing armor, combine armed attacks with unarmed strikes (not necessarily in the same round, though), etc. But, eventually your PC will start feeling that his armor is limiting him, preventing him from escaping blows that using his technique he can now avoid - then he may consider giving up on wearing armor. Same goes for unarmed strikes - have him hold on to his dagger, but occasionally use a flurry of blows⁽¹⁾ using a fist and an elbow of his other, empty hand - and let him realize that his hands have become faster and more accurate and than any weapon he is used to. This can lead him to relinquish the use of melee weapons, to use an interchanging combination of weapon and unarmed strikes or perhaps even seek "better" weapons which are more compatible with his developing abilities.

Bottom line - just because the "metagame concept" says your PC can now do something, doesn't automatically means he is aware of that and trusts himself enough to give up his old ways completely and immediately.

Or, he can find written records of a technique

Taken from wikipedia's entry on the Northern Praying Mantis kung fu style:

Origins

There are many legends surrounding the creation of Northern Praying Mantis boxing. One legend attributes the creation of Mantis fist to the Song Dynasty when Abbot Fu Ju [...] supposedly invited Wang Lang and seventeen other masters to come and improve the martial arts of Shaolin.
The Abbot recorded all of the techniques in a manual called the Mishou ("Secret Hands")
[...]
This manual supposedly disappeared until the Qianlong reign era,
[...]
The manual records Wang Lang "absorbed and equalized all previous techniques" learned from the 17 other masters.

Aside from self mastering a technique, your rogue may find a documentation of a certain technique - this can come instead of having him invent it from scratch, or as a latter supplement giving him more inspiration and advanced abilities.

At either case, you can probably work the narrative details with your GM so they won't clash terribly with the campaign settings (assuming that having a monk in it doesn't do that in the first place).

(1) Or whatever equivalent term used in 5e for that...

There is no RAW way to combine Sneak Attack with Unarmed Strikes

In order to allow this, you'd need to either waive the Sneak Attack requirement for a Finesse weapon, or create a homebrew item that work as a finesse weapon but still an unarmed strike.

Even Unearthed Arcana has the major warning that it is not tuned for multiclassing, but once you start multiclassing with UA or Homebrew, you enter unprepared territory. It's up to the DM and they're table to decide how they want things to work. Allowing certain interactions (i.e. if the spiked glove is considered a finesse weapon), may break certain other things (i.e. rogue sneak attack.) There are generally trade-offs for making decisions and if those tradeoffs disappear because of homebrew, then you may end up with a character who is more powerful than they should be.

The dangers of allowing this interaction

But in the end, it's up to the DM decide if those interactions break their game. Sneak attack was purposefully made to NOT integrate unarmed strikes. If you have a way to turn an unarmed strike into an armed strike, it may cause a problem. If you don't think it does, or you can handle it by increasing enemy difficulty, etc, then go for it.

A note on unarmed strikes(from guildsbounty and Miatog)

It is important to note that monk unarmed strikes are not necessarily nor are they always a punch. The strike can come from an elbow, knee, head, etc - which is why the Monk doesn't need a free hand to make use of their unarmed strikes.

In addition, once you add a finesse weapon to allow Sneak Attack, you are no longer making an unarmed strike. You have armed yourself with a finesse weapon.