slang (orig. and chiefly U.S.). An undercover agent; a spy.
1942 BERREY & VAN DEN BARK Amer. Thes. Slang §458/16 ‘Spotter.’ (One who spys upon employees.).. Silent eye, spook, spotter. Ibid. §765/7 Rat, rubber heel, spook, spotter, a person employed to detect irregularities. 1954 People (Austral.) 3 Nov. 24/1 The spooks were senior constables who wore no uniform, worked in pairs and followed constables about the city and suburbs to see if they did their work properly. 1961 John o' London's 20 Apr. 434/1 The idea of making a living as a spy ‘spook’ in current Washington slang is repugnant to most of us. 1966 R. THOMAS Spy in Vodka (1967) vi. 50 I'd like him to get out of the spook business. 1979 L. PRYOR Viper i. 9 ‘My training was also in espionage at the CIA farm.’.. ‘A spook,’ I said in wonder.
This suggests the word would be mainly used in reference to American spies, but I think the 1954 reference is Australians talking about constables, so at least occasionally it has referred to any undercover person.
Not much information on why exactly the word was chosen, but it probably was an easy jump from "ghost that haunts people" to "mysterious secret agent who spies on people".
Since it's slang, it's going to be hard to pin down the exact origin of the phrase. I can think of two plausible derivations:
The skin of a bruised eye is going to be swollen and pulled taut, and thus more reflective and "shiny" than the surrounding skin. I think this is secondary.
The most plausible metaphor: Shoe shiners are not as common now as they were in the early 20th century. "Shiner" could refer to how the unfortunate person hasn't gotten their shoe shined, (usually with black shoe polish,) but rather the eye.
Best Answer
According to Etymonline, the term probably derives from jack or jakes (regardless of the John Harrington angle).
Even the article mentioned by @ect says pretty much the same thing: