The toast goes back more than half a century before the scriptwriters of Casablanca used it in 1942.
From Anonymous, A Holiday Skip to the Far West (1884), we have this scene set in the Southern Hotel in St. Louis, Missouri:
"Any friend of Mr. O'Sullivan welcome. Will you nominate your poisons, gentlemen?" Since poisoning was apparently a necessity, I nominated whisky and water, in preference to a native mixture. It would be more satisfactory to know the cause of death, thought I, should this uncalled-for fatality occur. "Wall, —— here's looking at you," says my newly-acquired friend of the white apron, winking over the bottle-nose, and emptying his glass. "Here's how," responds the porter, and the curious bacchanalian ceremony came to a close. I am not proud, and enjoyed the hospitality, as the porter paid the drinks, be it said to his credit.
From "Things Pleasant and Otherwise," in Ballou's Monthly Mahgazine (May 1884):
"Ah," said the colonel, as he threw out a chew of tobacco, and took the bottle. "Here's looking at you."
"Drink hearty," the young man replied, and taking the bottle he held it up, and added, "Here's to the hair of your head."
From "Editor's Table" in The Yale Literary Magazine (April 1895):
Are your glasses filled, gentlemen? Then, here's to the rose of the girlhood—Dorothy, and one more in which that jolly devil, Saint Elihu, shall join for all his shadowy background and his austere countenance. And so fellow Collegians—"here's looking at you."
From Edith Chase & W.E.P. French, Waes Hael: A Collection of Toasts Crisp and Well Buttered (1903):
New Mexico
Cactus and prickly pear and sagebrush,/“Men with the bark on” and hearts brave and strong,/New Mexico, here's luck, here's looking at you;/You're a deep note in freedom's world-wide song!
Early extensions of the toast include this one from The Harrow (1889) [result not shown in snippet window]:
French returns in a few minutes and finds the Sophomore has cultivated it up. Waldo, at Hillsdale— "Here's looking at you." George — "Yes, through a glass darkly."
And this one from Wilbur Nesbit, The Loving Cup: Original Toasts by Original Folks (1909):
To you, and you, good fellows of the quill,/Belongs the Homage of the Wine I spill;/Here's looking at you from the Rosy Side,/And may you never lose a Dollar Bill.
I imagine that in a heavy-drinking, toast-oriented social environment, few dedications would come to the befogged mind more readily than "Here's looking at you," since the person addressing the toast is in all likelihood looking at the person thus addressed.
An early filmed instance of the expression occurs in the movie Three on a Match (1932), as Michael Loftus (Lyle Talbot's character) attempts to woo/seduce Vivian Kirkwood (Ann Dvorak):
Michael: Here you are, darling. [He hands Vivian a martini.] Well, here's looking at you.
Vivian: At me?
Michael: Yeah! And liking it, too!
Evidently, the expression was widely viewed as being quintessentially American, which would make sense in Casablanca as another marker of Rick's Americanness. At any rate, in one of J.P. Marquand's Mr. Moto stories (by 1938), we have this bit of dialogue [combined snippets]:
"Boy!" called Uncle William. "Whisky soda. ... Here's to you, Mr. Moto." Mr. Moto laughed again.
"Here is looking at you, gentlemen," he said. "That is the American expression, is it not? What beautiful weather we are having!"
"Yes," said Uncle William.
UPDATE: 'Here's Xing' as an American form
Because so many readers have echoed the comment/question (from andy256 below) "Yes, but ... what does it mean? Or if it is quintessentially American, what does that mean about America?" I decided to look into the idiomatic use of "Here's Xing" in U.S. usage.
Since Americans are (as Fowler says of people who grew up with the distinction between will and shall) "to the manner born," the form "Here's looking at you" doesn't strike us as particularly odd. Indeed, the similarly formed expression "Here's hoping" is used not just as the start of a toast, but as an equivalent (in a multitude of settings) to "I hope so."
One of the earliest examples of "here's hoping" in a Google Books search appears in "Metaphors of the People," in The Galaxy (May 1870), which quotes Mark Twain's description of toasts given in various locales in the U.S. West:
The reaching of the popular mind after odd fancies is illustrated in Mark Twain's humorous explanation of the etiquette of the bar-room on the Pacific coast. He says:
In a mining camp in California, when a man tenders you a "smile," or invites you to take a "blister," it is etiquette to say, "Here's hoping your dirt will pan out gay." In Washoe [Nevada], where you are requested to "put in a blast," or invited to take your "regular poison," etiquette admonishes you to touch glasses and say, "Here's hoping you'll strike it rich in the lower level." And in Honolulu, when your friend the whaler asks you to take "fid" with him, it is simply etiquette to say, "Here's eighteen hundred barrels, old salt." But "drink hearty" is universal. ...
The sentimental method of asking a person to drink is in the formula, "Suppose we shed a tear." The operation, strange as it may seem, is identical with "taking a smile." There is a frequent toast in some places which seems to contain considerable truth, viz.: "Well, here's another nail in my coffin."
Of all these expressions, only the "Here's another nail" and the "Here's eighteen hundred barrels" follow (arguably) grammatical form, but it appears that the "Here's Xing" wording arose fairly early and has shown impressive cultural persistence. In fact, it appears in James Fenimore Cooper's The Pioneers (1823):
Toasts were uniformly drunk ; and occasionally, some one, who conceived himself peculiarly endowed by nature to shine in the way of wit, would attempt some such sentiment as "hoping that he" who treated "might make a better man than his father; or "live till all his friends wished him dead;" while the more humble pot-companion contented himself by saying, with a most imposing gravity in his air, "come, here's luck," or by expressing some other equally comprehensive desire. In every instance, the veteran landlord was requested to imitate the custom of the cupbearers to kings, and taste the liquor he presented, by the invitation of "after you is manners;" with which request he ordinarily complied, by wetting his lips, first expressing the wish of "here's hoping," leaving it to the imagination of the hearers to fill the vacuum by whatever good each thought most desirable.
Given decades of "Here's hoping..." in the context of toasts and in the ordinary affairs of life outside barrooms, the emergence of "Here's looking at you" can hardly have struck listeners as a strange innovation. That toast, in turn, may have prompted the toast, "Here's mud in your eye," which a Google Books search finds as far back as 1912 in the form, "Here's mud in your eye and a quick trip back to Texas!" There are numerous competing interpretations of that expression, however.
Best Answer
fanny is defined in Collins English Dictionary at CollinsDictionary.com as
It is used frequently to describe someone in a not very polite manner
'Taboo' seems a little strong for fanny as it can be used in a relatively friendly manner, particularly in the phrase fannying about. Collins notes that dick used to be 'taboo' but no longer is. Dick can also be used in the same way: dicking about.
Using fanny as a verb, still carries all the baggage and you need to be cautious with your usage.
Amongst friends stop fannying about means no more than stop messing about, however when used aggressively the meaning goes back to the original and becomes stop messing about, you fanny. Calling someone a fanny aggressively is only slightly less offensive than calling them a cunt.
You can use it self-referentially of course.
but you wouldn't use that as an apology for arriving late for an interview.
Given the opportunity for someone to misunderstand the intention of the word fanny in fannying about I would avoid it in all situations where you are talking to people you do not know very well. It's generally OK with your friends (if they accept it) but it would not be the sort of phrase you'd use in a more formal setting.
Do bear in mind that both Dick and Fanny are still fairly common names in the UK, so if you hear someone using the word dick or fanny you need to comprehend what they mean. It's quite possible to say "I met a Dick/Fanny last night" meaning "I met someone, whose name is Dick/Fanny, last night" or it could mean "I met someone, who I really didn't like, last night"
Other 'messing aboutery' terms, listed in the order of what I consider least likely to offend to most likely to offend, the first three being not offensive at all.
[i] bumming about is the US meaning of bum (vagrant, loafer, (UK:tramp)) and means much the same - 'hanging about doing nothing much'.
[ii] rarely used in a friendly way.