Your PC can invent a new technique on his own
Taken from wikipedia's entry on the Southern Praying Mantis kung fu style:
Praying Mantis
The association of the term "Praying Mantis" with the style is also controversial. Each branch of the style offers a different explanation.
The traditions of the Chow Gar and Kwong Sai Jook Lum branches each maintain that their respective founders Chow Ah-Nam and Som Dot created their styles after witnessing a praying mantis fight and defeat a bird. Such inspiration is a recurring motif in the Chinese martial arts and can be found in the legends of Northern Praying Mantis, both White Crane styles, T'ai chi ch'uan, and Wing Chun.
If your rogue is observant enough - he may become inspired by some natural phenomenon, gaining insights about efficient and effective motions useful for offense, defense and maneuvering.
He doesn't need to be an acetic philosopher (though he may grow to become one as he advances) - he just needs the conviction and self-discipline to push him to go through the physical training required to hone his technique.
If your group's play style allows placing more emphasis on narrative at the cost of poorer combat optimization, you can even select to make the transition of your PC's reliance on his new martial skills gradual - continue wearing armor, combine armed attacks with unarmed strikes (not necessarily in the same round, though), etc. But, eventually your PC will start feeling that his armor is limiting him, preventing him from escaping blows that using his technique he can now avoid - then he may consider giving up on wearing armor. Same goes for unarmed strikes - have him hold on to his dagger, but occasionally use a flurry of blows(1) using a fist and an elbow of his other, empty hand - and let him realize that his hands have become faster and more accurate and than any weapon he is used to. This can lead him to relinquish the use of melee weapons, to use an interchanging combination of weapon and unarmed strikes or perhaps even seek "better" weapons which are more compatible with his developing abilities.
Bottom line - just because the "metagame concept" says your PC can now do something, doesn't automatically means he is aware of that and trusts himself enough to give up his old ways completely and immediately.
Or, he can find written records of a technique
Taken from wikipedia's entry on the Northern Praying Mantis kung fu style:
Origins
There are many legends surrounding the creation of Northern Praying Mantis boxing. One legend attributes the creation of Mantis fist to the Song Dynasty when Abbot Fu Ju [...] supposedly invited Wang Lang and seventeen other masters to come and improve the martial arts of Shaolin.
The Abbot recorded all of the techniques in a manual called the Mishou ("Secret Hands")
[...]
This manual supposedly disappeared until the Qianlong reign era,
[...]
The manual records Wang Lang "absorbed and equalized all previous techniques" learned from the 17 other masters.
Aside from self mastering a technique, your rogue may find a documentation of a certain technique - this can come instead of having him invent it from scratch, or as a latter supplement giving him more inspiration and advanced abilities.
At either case, you can probably work the narrative details with your GM so they won't clash terribly with the campaign settings (assuming that having a monk in it doesn't do that in the first place).
(1) Or whatever equivalent term used in 5e for that...
I play an alchemist myself. Here's how I did it:
Choose the Guild Artisan background. Select Alchemist as your Guild Business. Select Alchemist Tools as your tool proficiency.
Play a wizard, and specialize in Transmutation. Use the Minor Alchemy ability whenever appropriate — crafting an iron weapon into a silver one can be handy through the lower levels, even if it’s just a dagger to throw.
Choose some spells with an alchemical flavor. Acid spells work well. For a little added flair, you can refer to cold spells as “endothermic,” fire spells as "exothermic," etc.
Spend some downtime making acid, alchemist’s fire, or other chemical goodies for the party.
Put alchemy-related things into your character traits. My wizard’s personality traits include delighting in making batches of an “endothermic dairy confection” (ice cream) to amaze the townsfolk and celebrate his victories.
Best Answer
There is no non-metagaming way...
...because class is a metagame construct: from the characters' perspective, it doesn't exist. There is no good way to determine "class" as a hard fact for the character, because a particular set of abilities does not cleanly map to the character's identity and societal position in-world.
To put it a different way (using D&D 5E terminology)...
You have a two characters who wear heavy armor without discomfort, swing a greatsword with skill, and call on the powers of a deity to enhance their abilities and destroy their enemies. Are you dealing with a War Cleric or a Paladin?
You have two characters who wear medium armor, wield a longsword, and cast arcane spells1. Are you dealing with an Eldritch Knight Fighter, or a multi-classed Fighter/Wizard?
To emphasize the difference even more... all four2 of these characters, if asked in game (without metagaming), may call themselves a knight.
Spellcasting
Even within the same class, knowing a character's class may not tell you anything substantial about their capabilities. Knowledge that a character is a sorcerer, for instance, doesn't tell you anything about how the character might fight. Even if one knows a character is a sorcerer, without observation or simply asking in-character (i.e. "Hey, finger wiggler, what spells do you know?"), there is no question that would give insight on how a sorcerer is going to fight that isn't also asking for metagame information.
NPCs
Furthermore, in some editions, NPCs are built entirely differently from characters - they don't have classes or levels at all. In those cases, the name of the stat block in use may be more descriptive than a class, but it's not guaranteed to have anything to do with the character's role or place in society.
To use another D&D 5E example, using the bandit stat block does not mean the character is a criminal. It could just as easily be a poorly equipped town guard or a mediocre hunter/trapper.
TLDR
The point is this:
A character's class defines what the character can do, not who they are. The in-character answers may have very little bearing on the metagame/mechanical answer.
1We could go even further if it's a Githyanki we're talking about; they get the appropriate proficiencies via race. We could be talking to a pure wizard, bard, sorcerer, or warlock.
2...or five...