As Cerberus wrote about 'U' English, replying to "How do you do" with "How do you do" used to be the case among some classes in England (at least), but it seems to be (sadly) nearly extinct. Kate Fox writes in the first chapter of Watching the English (which is about talking about the weather):
We used to have another option, at least for some social situations, but the ‘How do you do?’ greeting (to which the apparently ludicrous correct response is to repeat the question back ‘How do you do?’) is now regarded by many as somewhat archaic, and is no longer the universal standard greeting. The ‘Nice day, isn’t it?’ exchange must, however, be understood in the same light, and not taken literally: ‘How do you do?’ is not a real question about health or well-being, and ‘Nice day, isn’t it?’ is not a real question about the weather.
So in this usage, "How do you do?" wasn't an actual question about the person: it was just a meaningless greeting, and for instance what one might say when introduced to someone (in lieu of "Pleased to meet you"), presumably while tipping one's hat. In reply, the other person, also wishing to make the same meaningless greeting, would say "How do you do".
Note that "How do you do" in this usage was even spoken as a statement, not as a question (i.e., without a rising tone at the end).
These days "How do you do?" is more likely to be interpreted as an actual question. Interestingly, as n0nChun observes, one does sometimes hear a similar exchange these days, with "What's up" getting the reply "What's up", or even just "sup" — "sup".
Best Answer
Nice, in its most common sense as an adjective means, either
1) Giving pleasure or satisfaction 2) Being kind or generous. Humble.
So when somebody asks you, "How are you?", then would you reply to them, something like this,
"Hi, I am very pleasure giving!" or "I am kind and generous!"
No, right? Hence, it is incorrect to use nice as an answer here. Whereas, fine can also be used to mean, "in good health", which is an appropriate response.