Whiterun Halls of the Dead Catacombs (in the first room, go down the left stairs and it will be to your left)
Mistveil Keep in Riften (Jarl's room)
Proudspire Manor in Solitude (has to be purchased for 25.000 gold)
College of Winterhold (Arch-Mages room)
Dragonsreach in Whiterun (Jarl's room)
Jorrvaskr in Whiterun (Kodlak's room)
Dead Crone Rock (On the table near the 'boss')
House of Clan Shatter-Shield in Windhelm
Black-Briar Lodge east-northeast of Riften (Second floor bedroom, next to bed)
Ansilvund Burial Chambers (northeast of Riften)
Wuunferth's Room in the Palace of Kings 2nd floor (Windhelm)
Stony Creek Cave (South of the Eastmarch Imperial Camp, last room on the table)
Rannveig's Fast south of Morthal, northeast of Rorikstead (after falling down the trapdoor in fron of the chest and dragon-word-wall)
Fellglow Keep northeast of Whiterun (straight up the stairs and to the right)
Dainty Sload (A ship along the coast southwest of the Solitude Lighthouse)
Sunderstone Gorge (Right of the alchemy on a table with skeletons)
Solitude Palace (Queen's room)
Yngvild, between Dawnstar and Winterhold (Very last room near the end)
Treasury House in Markarth (End table in Thonar's bedroom, which is the left from the front door)
Dwemer Museum in Markarth, entrance to the right as you enter Understone Keep (Table in room to the Left)
Thalmor Embassy (Elenwen's Solar, second floor, first bedroom on your left) (Or in the cave through which you escape, after jumping from the ledge, on a small table beside a skill book and dead mage.)
Hob's Fall Cave (Near the alchemy set on the left)
Pinewatch - House Northwest of Helgen with a Bandit hideout underground. Gem is in a storeroom before the 'boss'.
Now there are also mods that place map markers for them or otherwise make collection easier:
Best Answer
Skyrim puts its saves in the following directory in Windows Vista & Windows 7 and Windows 8:
Note that on Windows 8, the saves folder could be hidden, so you'll have to show the hidden files to be able to see it.
Skyrim puts saves in the following directory in Windows XP:
These files are not dependent on anything so you can freely share them across computers, delete, copy, backup, or anything else.