Does using a sword and spell count as dual wielding

If you play with a controller it is mapped correctly (right trigger controls right hand). There is no way to just map dual wielding on the PC

I was wondering if by attacking bare handed, you increase your 1-hand (or 2-hands) weapon skill mastery.

I played through the start of the game unarmed (Khajit), and never increased my one-handed or two-handed skills.

[...]do the Khajiits's +15 claw bonus mean that every one-hand weapon below damage 15 is useless?

If only comparing pure damage, yes.

Does, as example "Backstab" skill bonus apply just to any 1-hand weapons, or to barehand too?

The sneak perk Backstab applies to one-handed weapons, which bare-handed is not.

Can I, say, do a stealthy istant kill from behind when unarmed? Like breaking the neck maybe?

You can sneak attack while unarmed (and with a two-handed weapon) for extra damage. If you do enough damage to kill the opponent in one shot, you may be treated to a special animation. IIRC, there's an unarmed neck snap one.