Learn English – Losing bottles and bottling out

OED has this pre-dating Shakespeare:

m. to tear (†rend) one's hair, i.e. as a symptom of passionate grief.
1548 Hall's Vnion: Henry IV f. xiiijv, This knight..sobbed, wept, and rent his heare.
1609 Shakespeare Troilus & Cressida iv. iii. 33 Teare my bright haire, & scratch my praised cheekes.

It is something which can actually be witnessed in more demonstrative cultures. Mourners can indeed pull their own hair.

As an indication of exasperation, however ("I was pulling my hair out trying to get the copier to work!") there seems to be rather less information — although Shyam's trichotillomania may well be related.

The copulation sense of shag certainly seems to have come first, so to speak. From Francis Grose, A Classical Dictionary of the Vulgar Tongue, Third Edition (1796):

TO SHAG. To copulate. He is but bad shag; he is no able woman's man.

Eric Partridge, A Dictionary of Slang and Unconventional English, Fifth Edition (1961) sees the same possible connection that you do:

shag, v.t. To coit (with a woman) : late C. 18–20. Very gen. among soldiers in G.W. Grose, 2nd ed. Prob. ex [obsolete] shag, to shake,toss about. ...2. Whence, perhaps, v.i., to masturbate : Public Schools : certainly ca. 1900 and prob. many years earlier.

shag, adj. Exhausted, esp. after games : Marlborough College : C. 20. Perhaps ex shag, v., 2, q.v. (A thin and weedy dog that, ca. 1919–23, haunted the college precincts, was known as Shagpak or Shaghat, as Mr. A.B.R. Fairclough, formerly of the Alcuin Press, tells me.)

Interestingly, Partridge also has an entry for "wet as a shag" (from circa 1830), meaning "very wet indeed," in reference to the cormorant (or shag), whose feathers lack the oil that many swimming birds possess and thus get very wet indeed. This also explains the cormorant's habit of standing on a dock, rock, or tree branch with its wings motionlessly outstretched, to dry its feathers. I'm a bit surprised that Partridge didn't entertain the possibility that the exhausted sense of shagged might derive from the bird's motionless torpor at such times, following a vigorous session of fish catching.

John Ayto & John Simpson, The Oxford Dictionary of Modern Slang (1992) takes a view shag and shagged that is similar to Partridge's:

shag (1) coarse. verb trans and intr. 1 To have sex (with) 1788– ... 2 Used in curses and exclamations 1933– ... 3 An act of copulation. 1937– 4 One who copulates; used as a general term of abuse. 1971– ... [Origin uncertain; perh. from obs. shag to shake, waggle.]

...

shagged adjective Weary, exhausted; often followed by out. 1932– ...{Origin uncertain; perh. related to SHAG (1) verb.]

Ayto & Simpson also notes the existence of a verb shag from 1851 meaning "to wander aimlessly; to traipse; to go away." That term's origin is unknown.