According to Eric Partridge's A dictionary of slang and unconventional English, Hong Kong dog is a tropical fever, and the term is originally 20th century Royal Navy slang.
His source for this is 'Taffrail', or Capt. H Taprell Dorling, DSO, RN, in Carry On, 1916, especially the article 'the Language of the Navy, originally published not later than 1915.
The term is comparable with Malta dog, another local name for traveller's diarrhoea, is also from Royal Navy sailors.
Hong Kong dog can be found as far back as 1899. The Philadelphia medical journal says:
Sprue (psilosis linguae, Hong-Kong dog, Ceylon sore mouth, etc.) is a chronic catarrh of the alimentary canal from the mouth to the anus, accompanied by tenderness of the tongue, diarrhea of a special character, and an atony and ...
Also from 1899, Madam Izàn: a tourist story says:
Cholera at Port Said, leprosy at Colombo — I heard of a family who had caught it in their washed linen — the plague here. Then the fog. And besides that, a fever they call the Hong Kong dog, which is nearly as bad as the plague
William Ernest Russell Martin's 1924 The adventures of a naval paymaster has a whole chapter on Hong Kong dog, unfortunately not readable via Google Books.
But why dog? According to slang lexicographer and author of Green's Dictionary of Slang:
The primary divisions are place-related, images of dancing or fast movement [e.g. Aztec two-step, Greek gallop, Rome runs, Tokyo trots], and rhyming slang usually based on 'the shits'.
As for dog, there's anecdotal sources describing either being "bitten" by the dog, or the onomatopoeic "barking" noises you made in corners when you were suffering from it.
At All Costs by Sam Moses describes Malta dog during WWII:
Sand flies flew out of the cracks in the limestone, carrying their fever. Cockroaches, bedbugs, and lice ruled. The "Malta Dog," a virulent form of dysentery, barked in the corners. The hunger never went away.
Band of Eagles by Frank Barnard also describes WWII pilots falling sick:
"Bitten at last by the Malta Dog"
Some forum postings agree:
The "Malta Dog," a virulent form of dysentery, When you got it you barked in the corners.
Malta dog - was what you were bitten by in flight the morning after a night in downtown Valetta in "the Gut", after consumption of copious quantities of Cisk or Hopleaf beer; or (God forbid), Farmers' Wife wine. Usually comprised of the runs interspersed with the odd projectile vomit, both accompanied by the "cheese-wire round the forehead" headache. If you had a good aim you could throw up out the beam lookout window of a Shackleton with little risk of blow-back. Thank the Lord those days are over.
Malta was the headquarters of the Royal Navy's Mediterranean Fleet from 1814 until the mid-1930s, so there was a large navy presence. This ties in with the Royal Navy roots in Partridge's etymology for Hong Kong dog.
According to the Phrase Finder:
It's modelled on the 17th-18th century phrase "that cock won't fight". In the days of cock-fighting, a cock that wouldn't fight when out into the pit was a natural metaphor for a plan or theory that simply wouldn't work. A similar sporting metaphor from horse-racing is used when we say that a plan or theory "isn't a runner".
source
You use the phrase to describe an idea which won't work. It's a prediction of failure.
Example:
Person A: I heard you are going to ask Jane out on Friday.
Person B: I was going to ask her out, but after I found out she's seeing someone, I decided not to. That dog won't hunt.
Best Answer
J.E. Lighter, Random House Historical Dictionary of American Slang (1994) confirms the gist of the American Heritage Dictionary's coverage of the phrase "dog it":
Early more-or-less literal meanings of 'dog it'
The earliest instance of "dogging it" that an Elephind newspaper database search finds carries the literal sense "set a dog on it," as we learn an account of monetary losses that the writer blames on poor fencing between a farm owned by a man who raised cows, pigs, and other animals, and the writer's corn farm, in "Fences," in the [Ohio County, Indiana] Rising Sun (June 4, 1836):
Another early meaning of "dog it" appears in Delia Bacon, "The Elizabethan Men of Letters" the Sydney [New South Wales] Morning Herald (September 17, 1857), where the clear sense is "behaving like a [hostile] dog":
This usage—in the longer form "dogging it at its heels"—is indistinguishable in effect from the modern idiom "snapping at its heels." Although this form of the expression may share in common with more-recent senses of "dogging it" the core allusion to "behaving like a dog," the particular canine behavior involved is evidently quite different. Moreover, the "it" attached to "dogging" has a particular object in mind, whereas in the later senses the "it" is nebulous.
A narrower sense of "dogging it"—"following [something] the way a dog would"—appears in "The Great Nebula in Orion," in the Alexandria [Virginia] Gazette (February 5, 1883):
Here, of course, the temptation to use "dogging it" is probably not unrelated to Sirius's identity as the Dog Star. Again the "it" is something very definite.
Early figurative meanings of 'dog it'
An intriguing figurative instance of "dog it" appears in an untitled item in the [Lincoln, Nebraska] Capital City Courier (December 19, 1891, where the full expression is "yellow dogging it":
It's hard to say exactly what "yellow dogging it" means as used here. I am inclined to read it as meaning something like "dragging it [the musical comedy] like a sorry yellow dog"—but that may not be accurate. The term "yellow dog" has a long record as a pejorative in U.S. English. J.S. Farmer & W.E. Henley, Slang and Its Analogues past and Present (1904) has this short but vague entry for it:
And Mitford Mathews, A Dictionary of American English on Historical Principles (1951) has this:
The sense of "big yellow dog under the wagon" seems to be "a thing of which very little is known, but about which it is not safe to make assumptions." Whatever the sense of "yellow dogging it may have been, the 1891 Capital City Courier instance of that expression is the only one that my Elephind searches uncovered.
Another intriguing but ambiguous instance occurs in "Dramatic Notes" in the [Washington D.C.] Morning Times (October 4, 1896):
There is no further context offered for the use of "dogging it" in this item.
[[UPDATE (July 20, 2018): Spurred by a comment by EL&U participant (and poster of this question) Colin, I checked to see whether dog might not have had a particular meaning in late-nineteenth-century show-biz argot—and evidently there was. From Don Wilmeth, The Language of American Popular Entertainment: A Glossary of Argot, Slang, and Terminology (1981) [combined snippets]:
The two newspaper examples reported in this section of my answer reflect exactly those old show-business senses of "dogging it." Thanks, Colin, for the excellent observation.]]
Early instances of 'dog it' in horse racing and other sports
The first sports reference to "dogging it" appears in "Ninety-Eight for Old Logan," in the San Francisco [California] Call (April 2, 1897), where the meaning seems to be "losing spirit," "accepting defeat," or "refusing to go all out":
The "verb form "dogged it" appears less than six months earlier in the same newspaper. From "Joe Terry Likes Muddy Going Now," in the San Francisco [California] Call (November 21, 1896):
Poor Moylan is accused of dogging it again ten weeks later in "Pat Murphy Was on His Mettle" in the San Francisco [California] Call (February 6, 1897):
The San Francisco Call contains nine additional instances of horses who "dogged it" within the next two years—on November 17, 1897, on December 19, 1897, on March 31, 1898, on April 2, 1898, on November 18 1898, on December, 2, 1898, on December 7, 1898, on January 24, 1899, and on January 28, 1899. Thereafter, other newspapers—such as the Kansas City [Missouri] Journal and the Omaha [Nebraska] Daily Bee—both carrying the same report on February 26, 1899, from horse races in New Orleans, Louisiana, began to include the term.
The first Elephind match in which a news story applies "dogging it," in the same sense as with horses, to a person is in "Washington Fared Well in Inter-City Boxing," in the Washington [D.C.] Times (February 17, 1906):
Conclusions
Figurative use of "dogging it" appears in newspapers beginning in 1891, but the sense of the phrase, as used in two early nonsporting contexts, does not appear to be related to the sporting sense. Instances of "dogging it" and "dogged it" in the now familiar sense of "not trying hard" or "didn't try hard" begin to appear in U.S. newspapers in 1897—and they are highly localized early on.
Elephind searches turn up a dozen unique instances of "dogging it" or "dogged it" between November 21, 1896, and January 28, 1899—all in the same newspaper (the San Francisco Call) and all in the context of racehorses that turned in poor performances relative to expectations or after starting out well—before it appears in a similar sporting context anywhere else.
It seems highly likely that the popularization of this particular slang or idiomatic expression originated with its repeated use in to horse-happy Call.