Most fortuitously for you, just a couple of days ago I stumbled upon a book that answers this and most any question one might have on the word fuck and its multitudinous derivatives — anyone who has the slightest bit curiosity about this subject would do well to check out Jesse Sheidlower's The F-word, a very accessible and fun book. In writing the book, the author had access to the Oxford English's Dictionary's archives — the most extensive and renowned of all dictionaries — so what he has to say is likely the final word on the matter.
Let's start at the beginning of your questions. First of all, the germ of the idea behind the word motherfucker is likely very old. I'd imagine that since the dawn of language more than 200,000 years ago, people have been insulting one another by attacking their mothers. To appreciate the thrust of the insult motherfucker doesn't require knowledge of the technology, or culture, or local history of any particular milieu; the logic behind it is virtually ageless, because it is undergirded by a fundamental evolutionary truth, instinctually apparent to every creature since sex first came about — and that truth is that a mother's fidelity is dear, to both her and her partner, and it is mostly unfavorable circumstances that serve to erode it.
That was a mouthful. Let's make it clearer. Insulting a mother's fidelity is so powerful because it lies at the nexus at many of the most fundamental issues that a successful marriage must negotiate — saying I fucked your mother is a many pronged insult that implies a large score of unpleasant things. Indeed, it suggests that:
Your father is foolish and dull. He has no idea, or no power, to keep his wife from another man. Your mother two-times him with impunity.
Your mother is a slut and a harlot. Not only does she cuckold your father, but she does so wantonly in marriage, and one implication is that she might too be busy getting some to adequately take care of you and your siblings.
You are illegitimate, and can receive no legitimate property, because your mother is not good enough to make it into your father's stable of wives; she's no better than a common whore. Because of her sluttiness, your warden of a father cannot be sure that you are his, and thus he will not chance any of his life's hard work on possibly another man's child, an evolutionary disaster if there ever was one.
Your father is pathetic, and unable to provide for you. His prospects are dim, and things are so bad in your household that your mother must turn her attentions to another man so that you and your siblings get enough to eat. Not only is she in his debt, but figuratively, so are you and your father.
Further reasoning along the same lines shows why an insult like father fucker or some such that would be nonsensical; a father's fidelity simply isn't as important to the health of a household as a mother's is. Unlike a mother, a father can simply work to support all of the fruits of his dalliances; in fact, as long as you are a legitimate son, a father fucker might be an epithet you look upon with equanimity. It suggests that your father is powerful, or rich enough to be able to enjoy such a position. The asymmetry arises from an iron law: A member of the male sex in desperate straits is likely to be less promiscuous, not more, because no female in need of property will accept him; a woman in desperate straits is likely to be more promiscuous, not less, because access to her sex is the only thing she has to sell.
Thus, it isn't hard to see why a motherfucker, someone who fucks mothers, would easily become a byword of opprobrium and contempt; to have a potential motherfucker walking about is to have an existential insult that threatens a man's very identity as a legitimate member of his household.
With that behind us, let's go to the etymology. Unfortunately, some of the etymologies that are given here are nonsensical, since as The F-Word clearly notes, the word fuck itself was not attested until the end of the 15th century. It clearly came into its own as a vulgar word in the 16th and 17th centuries, replacing the word swife. However, given the puritanical nature of publication in those days, was not recorded in a English dictionary till 1775, and only till 1891 were more than a couple of words and a few quotations jotted down about it, in John Farmer's and W.e. Henley's Dictionary of Slang and Its Analogues.
You see, that was the nature of vulgarities back then; they might prosper in spoken English but would be scarcely written down until the late 19th centuries. [The book tells an amusing anecdote of a defendant rebutting a slander judgment against him by claiming as late as 1846 that fuck, the word he slandered a woman with, was not an English word.] Thus, we cannot be sure that motherfucker, borne of fuck was not actually attested in speech by the dates we have first written attestation, and sadly enough they are the only products amenable to historical investigation.
In any case, here are the first few citations given for the earliest known definition, a despicable or contemptible person:
1918 Letter in Journal of American History LXXXI 1585: Your low-down Mother Fuckers can put a gun in our hands but who is able to take it out? [1918 in H. De Witt Bawdy Barrack-Room Ballads: The little red run he grew and grew/fucked his mother and sister too.] 1928 C. McKay Banjo 229: I've been made a fool of by many a skirt, but its the first time a mother-plugger done got me like this.
The word motherfucker then really took off in the Army in World Wars, giving the second sense, a fellow; a person; an admire person; a formidable person:
1958 Stack a Lee (typescript, Kinsey Institute) 1: He...said who put the hole in this motherfucker's head/Who could the murderer of this poor man 1964 R.D Abrahams Deep down in Jungle (Appendix II) 261: One of the best things which can be said of a man is that he is a "mean motherfucker" or a "tough motherfucker", but to call him just a "mother-fucker" is to invite reprisal.
And after the Wars, and the return of the troops, I'm supposing it exploded onto the popular consciousness as result of vanishing indecency standards in the '60s, and that was that.
Best Answer
It probably started in a different sport such as baseball.
For example, the 8 June 1906 St. Louis Post-Dispatch at page 16 says:
It would be more logical to originate in golf, where you could be "tied n up" or "tied n down". For example, from the 29 January 1963 Arizona Republic:
For an older golf example see the 16 July 1916 Houston Post:
For tennis, as far as scoring points, the earliest example I see is in the August 1939 Boys' Life at page 11:
The 1971 book Tennis attempts to stop people from saying "up":
However, the 1991 Tennis for Beginning and Intermediate Players says: