This link claims that one cannot be sure of origin of this phrase. Three explanations are given here, but they are not very convincing (I am not a native speaker).
In one of our newspapers, someone gives an explanation. Apparently,
Duke of Wellington, the man who defeated Napoleon Bonaparte in the
Battle of Waterloo in 1815, though quite the ladies man, wasn’t really
much to look at. The most prominent feature of his not-so-handsome
face was his rather long nose. The officers and the men who served
beneath him began to affectionately refer to any nose which was
unusually long as ‘duke’. With the passage of time, this word began to
be used to refer to any nose; of whatever size. Since human fists are
notorious to be employed in fights to put a ‘duke’ out of joints,
fists began to be called ‘duke buster’. Soon, the word ‘buster’ was
dropped, and everyone started referring to fists as ‘dukes’.
This sounds like a great story, but he did not cite any references. Since this explanation is so very different from many others available on-line, I wonder if there is some reliable reference in support of his claim.
Best Answer
Tom Dalzell’s The Routledge Dictionary of Modern American Slang and Unconventional English says the word dukes for fists in “put up your dukes” was attested as early as 1859. Dalzell says that forks was slang for fingers, and suggests that forks became dukes by way of the rhyming slang “Duke of Yorks”.
Credence is lent to this theory by Albert Barrère and Charles Godfrey Leland’s 1890 edition of A Dictionary of Slang, Jargon & Cant, which attests the very similar slang expression “put up your forks” meaning to challenge to a fight. (To put one’s forks down, however, was to pick a pocket!)
More corroboration comes from the 1859 The Vulgar Tongue: a Glossary of Slang, Cant, and Flash Words and Phrases by “Ducange Anglicus” (a wonderful pseudonym which loosely translates as “English dictionary”). “Anglicus” attests the verb fork in the phrase “fork out the tin” meaning “hand out the money”, used in London between 1839 and 1859. (Fork out is surely cognate with today’s idiom fork over.) This puts fork at the right place and time to make it possible for Cockney rhyming slang to invoke the noble Duke of York exactly as suggested by Dalzell.
However, the intermediate phrase “put up your Duke of Yorks” is theoretical: not attested in any text of which I am aware, which still leaves open the possibility of some alternative origin for the term.