While wielding the first weapon set, will the gems from the second weapon set level up at the same rate

path-of-exile

I think that all gems receive 10% of each amount of EXP your gain. But on the same wiki's page, you can read:

Gems gain experience and levels alongside the character wearing them.

And I've got the feeling that gems on my second weapon set (which I rarely use) don't level up as fast as the ones on my first weapon set… Any clue ?

Yes they should. You can read that from the same page you provided:

Gems equipped in both weapons slots gain experience, though only the currently active weapon slot's skills can be used.

It's probably that the gems you're using on the second set happen to need more experience points to level up.