Most rust-proofing paints are fairly hard enamels, even if they are matte finish. Unlike house paints, they are not supposed to chalk off. It is more likely that handling the matte finish might make it a bit shiny in places rather than the finish coming off on your hands.
There are some matte polyurethanes, but they may not be as hard as the finish you are already putting in it. There is an article here describing matte poly.
If you do try a poly over the enamel, do a test in an inconspicuous area first.
SUPPLEMENT:
It looks like Rustoleum makes a matte clear finish that could be simply spray painted over your other finish. Other brands may have similar products. I would still do a test in an inconspicuous place.
The texture difference is most likely the function of brush vs. roller. Do not thin the paint. Try using a roller, but use very litle paint on it. You are trying to get just a bit of a top coat that give the slight stippling that vitually all rollers leave behind.
Best Answer
It is generally best to stay with whatever base material is used for the first coat. That is, if the "special" paint was oil based, then continue with an oil based paint. Switching to say acrylic or water based will not provide a good bond between paints.
Can you use it... well you can, but you may need to use several coats and you may find it comes off easily when bumped with the vacuum cleaner etc.