OED
The Oxford English Dictionary defines far out as:
Of jazz: of the latest or most progressive kind. More generally, avant-garde, far-fetched; excellent, splendid. orig. U.S.
Their first quotation is an 8th November 1954 Time Magazine article entitled "Far-Out Words for Cats":
Jazz lingo becomes obsolescent almost as fast as it reaches the public ear.‥ A daring performance was ‘hot’, then ‘cool’, and now is ‘far out’.
1940s
We can see this term beginning to emerge in Simon Michael Bessie's 1938 book Jazz Journalism: The Story of the Tabloid Newspapers:
The third class, divertingly far out on the lunatic fringe, embraces such journalistic peep-shows as Bernarr Macfadden's New York Evening Graphic.
Whilst the book is about tabloid papers and not jazz, it shows "far out" being applied to something novel and possibly avant-garde.
Here's Esquire's 1946 Jazz Book applying it to jazz of the most progressive kind:
Much of its drive came from two other jazzmen on this list: drummer Davey Tough and bassist Chubby Jackson. Both are incomparable musicians; both have contributed mightily to that spark and zest which are so characteristic of Woody's band and which take it far out of the realm of the typical swing band.
Here's another snippet, from a 1947 The Jazz Record:
George [Lugg] sat staring miserably at the floor, and when the trombone solos came along he made impatient gestures and said: "That wasn't what I wanted to do at all." But in some of those ensemble passages, banging away behind Goodwin's trumpet and Scott's clarinet, with a good rhythm section pushing, he began to grin and nod his head. "That's it!" he'd say. 'That's good." He was so impersonal, so far out of himself, a musician listening to music and judging it by what he heard.
And from the next page:
Shuffle up that "Chain Gang" thing. Hear this cat blowing blue in the lost reaches of the world's last swamp. Go with this Yank into the eternal night, so far out of the world that nothing remains but the half-remembered crying of a horn. Born of slavery's misery comes jazz, bringing the gap from cat to cat with a blue bridge.
These may not be the exact use of the phrase, but give an indication of how it came around.
1954
It appears to have used as a set term in a publication of the South Dakota State University sometime in 1954:
... Ella Fitzgerald and hundreds of other "cats." To put expression for our word "darin;[?] into jazz terms, the jazz today is "far-out".
Billboard of 24 April 1954 reviews Les Elgart Orchestra's song "East is East":
A pretty instrumental with an exotic Oriental flavor. The virtuosity of Elgart's intrumentalists makes for easy listening, but there is a solid beat here for dancing. A little on the modern side, but not too far out.
Billboard of 26 June 1954 reviews Serena Shaw's "St. Louis Blues":
Here is the perennial as you might expect Yma Sumac to sing it. Miss Shaw, aided by exotic sounds from drums, organ and echo chamber, soars up and zooms down a two-octave range at a dizzying pace, working up to a frantic pitch at the end. It's pretty far out, but deejays might find it unusual programming.
The same November 8, 1954 Time Magazine cited by the OED had another article that used:
Says Jazz Promoter Norman Granz, who does not always understand [David] Brubeck's "far-out" music: "He's way out on Cloud 7."
I asked you which IE languages you know - in order to critically evaluate these three hypotheses, a strong background in the history of Latin (at least) is necessary. There are three major textbooks on the history of Latin - Baldi 1999, Sihler 1995, and Weiss 2009.
The first hypothesis is best supported by evidence - and, in fact, pretty standard now (for example, de Vaan 2008).
The Anlaut (word-initial) PIE *gwh> Lat. f sound correspondence is well documented, cf. Latin formus 'warm' - MnE warm; Greek thermos; Rus. zhar 'heat', goret' 'burn' etc. We still don't really know how PIE *gwh turned into Latin f (via *χw?) but this correspondence is regular.
We may ignore the perfectum fefelli because it's a relatively new coinage (double ll), cf. pello-pepuli, fero-tetuli (Meiser 1998), although reduplicated perfectum is usually archaic/rare in Latin.
The second "hypothesis" does not stand to scrutiny - supposedly, Latin fallo is derived from Latin facio. The person who came up with that hypothesis doesn't know Latin morphology at all. I don't know of any rule of Latin word-formation that could explain such a connection.
The third hypothesis does not have any explanation - it stops at Latin fallo.
Best Answer
The copulation sense of shag certainly seems to have come first, so to speak. From Francis Grose, A Classical Dictionary of the Vulgar Tongue, Third Edition (1796):
Eric Partridge, A Dictionary of Slang and Unconventional English, Fifth Edition (1961) sees the same possible connection that you do:
Interestingly, Partridge also has an entry for "wet as a shag" (from circa 1830), meaning "very wet indeed," in reference to the cormorant (or shag), whose feathers lack the oil that many swimming birds possess and thus get very wet indeed. This also explains the cormorant's habit of standing on a dock, rock, or tree branch with its wings motionlessly outstretched, to dry its feathers. I'm a bit surprised that Partridge didn't entertain the possibility that the exhausted sense of shagged might derive from the bird's motionless torpor at such times, following a vigorous session of fish catching.
John Ayto & John Simpson, The Oxford Dictionary of Modern Slang (1992) takes a view shag and shagged that is similar to Partridge's:
Ayto & Simpson also notes the existence of a verb shag from 1851 meaning "to wander aimlessly; to traipse; to go away." That term's origin is unknown.