In these sentences, "do they really" essentially means "is it really true that they". If you rewrite the sentences like that, it should become clear which is correct:
Is it really true that they have nothing in common?
Is it really true that they have anything in common?
Obviously the version with nothing is correct (the statement you are curious about is "They have nothing in common"); the version with anything doesn't make any sense.
But what you might be interested in is a similar question using anything:
Do they have anything in common?
This asks simply that; do they have anything in common? By itself, there is no implication that the speaker thinks they do or don't (but it's likely that intonation would make this clear.) Since your question referred to confirming an existing suspicion, you might consider a conversation like this.
Person A: Do they have anything in common?
Person B: They both know the sky is blue?
Person C: Oh, come on. Do they really have nothing in common?
You might also consider this common variant, as mentioned by StoneyB in comments:
Don't they have anything in common?
Note that anything is stressed, and don't is used rather than do. The implication is that the speaker is surprised to learn that the people supposedly don't have anything in common, and is checking that this assumption is true.
Best Answer
Your b) versions are ungrammatical in standard use because they employ double negatives.
If you want to use 'nothing' in these contexts you must delete the 'not':
Even in non-standard use, where double negatives are sometimes employed for additional emphasis, you are unlikely to encounter them in the forms you offer; the negated verbs would be contracted:
In my part of the country you wouldn't say even that; you'd say: