Early dictionary coverage of 'quim'
Francis Grose, A Classical Dictionary of the Vulgar Tongue (1785) has nine slang terms for "the private parts" of a girl or woman—to wit: bumbo, Carvel's ring, cauliflower, cock alley (or cock lane), commodity, madge, money, muff, and notch, plus an unidentified tenth one, ****, that appears in the entry for cauliflower. My guess is that **** does not stand for quim.
Pierce Egan's 1823 revision of Grose generally uses the term monosyllable in place of "private parts." This edition of the book removes several out-of-date terms from the 1785, and introduces fourteen new ones: black joke, bottomless pit, brown madam (or brown miss), Buckinger's boot, bun, dumb glutton, Eve's custom house, hat (or old hat), Miss Laycock, monosyllable (or venerable monosyllable), mother of all saints, tuzzy-muzzy, water-mill, and (at long last) quim. Here is Grose & Egan's entry for quim:
QUIM. The monosyllable: perhaps, from the Spanish quemar, to burn. (Cambridge.) A piece's furbelow.
This same definition, with "private parts of a woman" in place of "monosyllable" appears in Lexicon Balatronicum: A Dictionary of Buckish Slang, University Wit, and Pickpocket Eloquence (1811), which presents itself as Grose's original compilation "now considerably altered and enlarged, with the modern changes and improvements, by a member of the Whip Club." So unless Egan is the anonymous member, the inclusion of quim antedates his administration of Grose's dictionary.
John Jamison, Supplement to the Etymological Dictionary of the Scottish Language, volume 2 (1825) has entries for the (possibly unrelated) adjectives queem/quim and quim:
QUEEM, QUIM, adj. 1. Neat, fit, filled up to the general level, Upp. Lanarks., Ettr. For. [Cited example:] When the year grown auld brings water cauld,/We fle till our ha's sae queem. Marmaiden of Clyde, Edin[burgh] Mag[azine] May 1820. 2. Applied to what is made close and tight, ibid. 3. Calm smooth, Gall. [Citation omitted.] 4. Metaph. used, as conjoined with Cosh, to denote intimacy. [Example:] "It shall be observed, that they shall fall in more than ever, into an intimacy with the malignant enemies to the work of God, and grow quim and cosh with them while they are not only cold toward the truly tender, but cruel against them." McWard's Contend. p. 262 "Quim and Cosh, pliable and fit;" Gl. ibid. But this does not properly express the sense. The idea is evidently borrowed from joints that are exactly fitted, and adhere closely to one another.
...
QUIM, adj. Intimate. V. QUEEM.
Jamieson also notes, in connection with the verb "to Queem" (meaning "To fit exactly; as, to queem the mortice, or joint in wood"):
The O.E. v. to Queme, to please, to satisfy, is undoubtedly the same, used in a secondary or oblique sense ; because a thing is said to to please or satisfy, that fits our ideas or wishes.
As for cosh, volume 1 of Jamieson, Supplement to the Etymological Dictionary of the Scottish Language (1825) offers these entries:
COSH, adj. 1. Neat, snug. ... 2. Comfortable as including the idea of defence from cold, Ayrs. [Citation and etymological discussion omitted.]
COSH, adj. Denoting such a position that a hollow is left below an object, Galloway.
So "quim and cosh," in Scottish English in the early nineteenth century, seems to have meant something like "snug and tight." The possibility that this phrase made its way south and acquired sexual overtones is unsupported in Google Books search results. It's also somewhat problematic that, though quim is reported in dictionaries like the Lexicon Balatronicum, cosh is nowhere to be found in them.
Further complicating this etymological theory is the possibility that quim, in the form queme, goes back to at least the early seventeenth century. James Halliwell, A Dictionary of Archaic and Provincial Words, second edition, volume 2 (1852) has this entry for queme:
QUEME. (1) To please. (A.-S.) [Cited example from Lydgate omitted.] (2) To bequeath ; to leave by legacy. (3) The same as queint, q. v. [The entry for queint reads simply, "The pudendum muliebre."] "I tell you, Hodge, in sooth it was not cleane, it was as black as ever was Malkin's queme," Tumult, play dated 1613, Rawl. MS. Grose has quim, which he derives from the Spanish quemar, to burn. It is, perhaps, connected with the old word queint, which as I am informed by a correspondent at Newcastle, is still used in the North of England by the colliers and common people.
The complication here is that, according to Gordon Williams, A Dictionary of Sexual Language and Imagery in Shakespearean and Stuart Literature, volume 1 (1994), the instance that Halliwell cites remains an "untraced (OED) C17 example."
And finally, J.S. Farmer & W.E. Henley, Slang and Its Analogues, volume 5 (1902) identifies several variants of quim—queme, quimsy, quimbox, and quin—but cites only three examples: Halliwell's "old play" from 1613, a ballad from circa 1707 (see the next section below), and Halliwell's dictionary comment on queme.
Early recorded occurrences of 'quim' in the wild
The earliest Google Books match for quim in the relevant sense is in "The Harlot Un-mask'd," a ballad from circa 1707, reprinted in John Farmer, Merry Songs and Ballads Prior to the Year A.D. 1800, volume 4 (1897):
How happy the State does the Damsel possess?/Who would be no greater, nor can be no less:/On her Quim and herself depends for support:/And is better than all the Prime Ladies at Court:/What though she in Grogram and Lindsey does go/Nor boasts of gay Cloathing, to make a fine Show;/A Girl in this dress may be sweeter by far,/Than she that is stitch'd by a Garter and Star,/Than she that is, &c.
Tho' her Hands they are red, and her Bubbies are coarse,/Her Quim for all that, may be never the worse:/A Girl more polite with less Vigour may play,/And her Passion in Accents less charming convey:/What tho' a brisk fellow she sometimes may lack,/When warm with Desire, and stretch'd on her Back:/In this too great Ladies Example afford,/Who oft put a Footman in Room of a Lord,/Who oft put a Footman, &c.
John Farmer is J.S. Farmer, the coauthor of Slang & Its Analogues, and he isn't shy about calling a **** a ****, so he is very unlikely to have bowdlerized the wording of the ballad he quotes; nevertheless, he doesn't explain how he figured the date of the ballad as circa 1707.
Another relatively early English instance of quim—and the next one chronologically in Google Search results—may or may not be intended in the relevant sense. From greyhound races reported at Smee for November 12, in The Sporting Magazine: Or, Monthly Calendar (December 1795):
Mr. Hamond's (Mendham) Quim won ag[ain]st Mr. Holt's (Russel) Bacchus, 1 gui[nea].
People name their dogs (and horses) after various odd things, and The Sporting Magazine evidently didn't pursue the question of how Mr. Hemond's greyhound came by its name, so this instance seems scarcely worth mentioning, except for the extreme rareness of any mention of quim in English sources before 1800. The next Google Books match is to the Lexicon Balatronicum of 1811, which brings us full circle.
Conclusions
Because the slang term quim seems to have been considered extremely vulgar, it has left a very elusive trail in the historical record. An untraced sighting in a 1613 play (in the form queme) is the oldest claimed instance that I've encountered. And a double occurrence in a ballad dated to circa 1707 is the first confirmed instance of quim itself. But slang dictionaries report on quim starting in 1811, without adding significantly to the database of actual occurrences of the term. In Google Books search results, this state of affairs continues until roughly 1888, when the anonymously authored My Secret Life (which uses the term frequently) appears.
As for where the term came from, the Whip Club member in 1811 suggested the Spanish word quemar, to burn. Eric Partridge, Dictionary of Slang and Unconventional English, fifth edition (1961), nominates the Celtic cwm ("a cleft, a valley"). And there is of course the archaic English and Scottish English adjective quim (or queme or queem), which can mean pleasing, satisfying, gratifying, or the like.
Ultimately we're dealing with a slang word that may go back more than 400 years and appears to have been widely known near the beginning of eighteenth century—and certainly by the end of it—and yet has left few traces in the written record between 1600 and 1811. Under the circumstances, "origin unknown" seems a suitable (though regrettable) etymological conclusion to reach.
Best Answer
I don't think that yikes as an exclamation has any direct connection to yoicks or hoicks, or with yike (the cry of the green woodpecker of Britain and continental Europe, recorded starting in the late 1800s), or with the baby-talk word yikes meaning "likes" and popular in the late 1800s and early 1900s)—or for that matter with yikes in the eighteenth-century Polish/Yiddish sense of "family and academic background" or with Yike in the sense of a traditional Siamese theatrical performance described by nineteenth-century travelers.
Rather, I think that yikes emerged in the United States as a variant of the slightly older word yipes, which itself may have arisen in connection with "yip" and "yipe"—sounds that a dog makes. Though my view on this question is by no means incontestable, I base it on some circumstantial considerations that at least seem consistent with such a derivation.
Early 'yikes'
The strongest case I can make for the modern term yikes as a lineal descendant of yoicks—a call to one's hounds during a hunt—is that the two words appear together in Harry Hieover, Stable Talk and Table Talk, or Spectacles for Young Sportsmen (London, 1845):
But that's the last we hear of any similar exclamation in Google Books results until 1903, where yike arises as a response to a U.S. Army roll-call in Hamilton Higday, "A Day in the Regular Army," in The World's Work (New York, January 1903):
But neither the English fox-hunt yikes from 1845 nor the American roll-call yike from 1903 seems to have inspired any similar expressions in the Google Book database in the decades immediately following their appearance. In fact, the next slang use of yike to appear in the results appears in a 1943 issue of American Speech in the context of Australian slang used to describe a ticket scalper outside a sporting event who initially suspects that two ordinary citizens are plainclothes policemen [combined snippets]:
Here yike seems to refer to the game or sporting event or (perhaps) prize fight at which the scalper is trying to resell his "briefs" (tickets). I found only one other instance of Australian slang usage of yike, in Thomas Hungerford, Riverslake (1953); the meaning there seems to be a fight or tussle.
'Yikes' in the modern sense
Google Books search results show a number of matches for yikes beginning in the late 1940s. The first of these is from a 1947 U.S. Army pamphlet that appears to be an updated version of "Army Editors' Manual" (1943), a guide to writing and editing news stories for Army publications [combined snippets]:
A flurry of matches for yikes occurs in the period from 1949 to 1951, in the unmistakably modern sense of "wow," "zowie," or "uh-oh." From Robert A. Heinlein, The Man Who Sold the Moon (written in 1949 but published in 1951, according to Wikipedia):
From Ellery Queen, Double, Double: A New Novel of Wrightsville (1950) [snippet]:
From May Wallace, A Race for Bill (1951):
And from Robert A. Heinlein (again), Between Planets (1951) [combined snippets]:
As these examples suggest, yikes showed up in print through the backdoor of literature—science fiction, murder mysteries, and kid fiction (A Race for Bill is a soapbox derby novel). By 1953, yikes had also appeared in two books from the Happy Hollisters series and in Boys' Life magazine.
The 'yipes' connection
So far I've focused on yike and yikes, and on the emergence of yikes in its modern exclamatory sense in the late 1940s. But there is an obvious precedent for yikes in the word yipes—used in precisely the same way as yikes but beginning approximately a decade earlier. From Good Housekeeping, volume 113 (1941):
From The Saturday Evening Post, volume 216 (1943):
From Western Aerospace, volume 23 (1943):
From Marine News, volume 31 (1944) [combined snippets]:
And from Phantom Lady, volume 13 (1947) [a comic book]:
Possible sources of 'yipes' and 'yiping'
So where did yipes come from? One plausible source is dog sounds. From Katherine Milhous, Herodia: The Lovely Puppet (1942):
From Coulton Waugh, The Comics (1947):
A Google Books search for yipe turns up some considerably older matches. From G.E. Foster, "The Tail of a Dog," in The [Anamosa, Iowa] Reformatory Press (October 3, 1908):
From The Western Honey Bee, volume 2 (1914):
From The Editor & Publisher and The Journalist (December 18, 1915):
From P.G. Wodehouse, "Piccadilly Jim," in The Saturday Evening Post (November 11, 1916):
This usage may have led to the attribution of metaphorical yiping to human beings. For example, from Joel Blanc, "Firm of Henderson & Son," in National Hardware Bulletin (November 1913):
And from George Somerville, The Boardwalk Love Letters of Hiram and Ella (1915):
And from "Mexico and Intervention," in American Blacksmith Auto & Tractor Shop (February 1920):
And from The Garment Worker: Official Journal of the United Garment Worker of America (October 14, 1921):
Unlikely alternative derivations
But there are other possibilities as well.
From Max Schoen, "Music and Educational Tendencies of the Day," in Education (November 1917):
The source of yipes here might be dog sounds, but it might just be the onomatopoeic sound that the author thought that the chorus made.
From Fred McMullen & Jack Evans, Out of the Jaws of Hunland: The Stories of Corporal Fred McMullen, Sniper, Private Jack Evans, Bomber, Canadian Soldiers (1918):
This episode of "Yipes" for "Ypres" seems unrelated to anything else in the Google Books results.
From Kay Boyle, My Next Bride (1934) [combined snippets]:
Again the yipes might resemble the yipping of dogs or they might be imitative of the laughing sounds themselves.
Conclusions
From the Google Books results I looked at, it seems most likely (though by no mean certain) that the modern exclamation yikes arose during the late 1940s as a variant of yipes, which had gained some popularity as an exclamation during the preceding decade, and which had in turn probably arisen from the earlier yipe—an onomatopoeic spelling of a dog's nervous or pained yelp, in use from the first decade of the twentieth century.