Various dictionaries have different things to say.
What price [fame/success/victory etc.]?
something that you say which means it is possible that the fame, success etc. that has been achieved was not worth all the suffering it has caused
What price victory when so many people have died to make it possible?
(Cambridge Idioms Dictionary, 2nd ed.)
price [...]
what price (something)? what are the chances of (something) happening now?
(Collins English Dictionary – Complete and Unabridged)
What price something?
What is the value of something?; What good is something? (Said when the value of the thing referred to is being diminished or ignored.)
Jane's best friend told us all about Jane's personal problems. What price friendship? Jack simply declared himself president of the political society. What price democracy?
(McGraw-Hill Dictionary of American Idioms and Phrasal Verbs)
what price ——? used to ask what has become of something or to suggest that something has or would become worthless : what price justice if he were allowed to go free?
(New Oxford American Dictionary 2nd edition, from OS X)
Apart from Collins's strange definition, it seems that the general meaning of "what price X?" is "what's the value of X?" (not "what's the price of X", in the modern sense of price).
As for how it came to be, my wild speculation of the day is that it could have been used grammatically in an anaphoric expression, eg What price is freedom to us if we tolerate this tyranny? What price justice? etc.
As other answers have pointed out, it refers to a situation in which it's pointless to try because it's impossible to win: either you're going to get a hiding, that is, take a (metaphorical) beating on your hide, or else...nothing. There is nothing else. You're going to lose, and you're probably going to lose so badly that your opponent is going to gain nothing from beating you.
Best Answer
It's likely derived from the idiom 'face/confront your demons': http://www.usingenglish.com/reference/idioms/face+your+demons.html
Here are some various instances of the phrase in books which show its relation to confronting fear or inner turmoil.
Lightness of Body and Mind: A Radical Approach to Weight and Wellness
Learning to Breathe Again
Sober Words
Mickey Newbury Crystal & Stone: Second Edition
The Itch: A Novel
Archangel's Storm
An Echo in the Silence: Selected Poems 1992-2002
However if we remove the possessive pronoun from the phrase so that it becomes dance with demons, the meaning changes entirely and becomes equivalent to dance with the devil phrase. Instead of meaning to overcome your fears and temptations to sin, it becomes the opposite: to succumb to your temptations.
Meditations on the Glory of Christ: Genesis through 2 Chronicles
Encyclopedia of witchcraft: the Western tradition, Volume 4
Dancing With Demons
SEER: He Danced with Demons. Can He Pay the Price?
The Sea Change: A True Tale of High Adventure
In the Shadow of the Moon: Prologue
Empress
Cataclysm
The worms
India, Ancient and Modern, Geographical, Historical, Political, Social, and ...
Women's Religions in the Greco-Roman World: A Sourcebook