The paragon stones and the paragon portal

From the Unofficial Elder Scrolls Pages' wiki article on standing stones in Skyrim:

The Apprentice Stone

• On an island in a river just to the west of the line joining Morthal and Solitude.
• Those under the sign of The Apprentice recover Magicka faster, but are more susceptible to Magicka damage.

The Atronach Stone

• South of Windhelm, in the southernmost part of the hotwater zone, between an axe shaped lake and a 'u' shaped lake (closer to this one).
• Those under the sign of The Atronach absorb a portion of incoming spell damage and have a larger pool of Magicka, but recover it more slowly.

The Lady Stone

• In a lake, on an island directly north of Falkreath
• Those under the sign of The Lady regenerate Health and Stamina more quickly.

The Lord Stone

• East of Morthal, up on the mountain.
• Those under the sign of The Lord are more resistant to both Magicka and physical damage.

The Lover Stone

• East of Markarth
• Those under the sign of The Lover always feel a Lover's Comfort. (All skills improve 15% faster.)

The Mage Stone

• Southwest of Riverwood
• Those under the sign of the Mage will learn all magic skills 20% faster.

The Ritual Stone

• East of Whiterun, North of Whiterun Stormcloak Camp
• Once a day, those under the sign of The Ritual can reanimate nearby corpses to fight for them.

The Serpent Stone

• Directly east of College of Winterhold
• Once a day, those under the sign of The Serpent can use a ranged paralyzing poison on opponents.

The Shadow Stone

• South of Riften
• Once a day, those under the sign of The Shadow can become invisible for an extended period.

The Steed Stone

• West of Solitude.
• Those under the sign of The Steed can carry more and do not suffer a movement penalty from armor.

The Thief Stone

• Southwest of Riverwood
• Those under the sign of the Thief will learn all stealth skills 20% faster.

The Tower Stone

• Directly west of College of Winterhold
• Those under the sign of The Tower have the option to automatically open Expert or lower locks.

The Warrior Stone

• Southwest of Riverwood
• Those under the sign of the Warrior will learn all combat skills 20% faster.

You're tasked with finding these for the "No Stone Unturned" quest.

From the UESP:

The stones are located in the following places:

• Dark Brotherhood Sanctuary (Astrid's room)
• Whiterun Halls of the Dead Catacombs (in the first room, go down the left stairs and it will be to your left)
• Mistveil Keep in Riften (Jarl's room)
• Proudspire Manor in Solitude (has to be purchased for 25.000 gold)
• College of Winterhold (Arch-Mages room)
• Dragonsreach in Whiterun (Jarl's room)
• Jorrvaskr in Whiterun (Kodlak's room)
• Dead Crone Rock (On the table near the 'boss')
• House of Clan Shatter-Shield in Windhelm
• Black-Briar Lodge east-northeast of Riften (Second floor bedroom, next to bed)
• Ansilvund Burial Chambers (northeast of Riften)
• Wuunferth's Room in the Palace of Kings 2nd floor (Windhelm)
• Stony Creek Cave (South of the Eastmarch Imperial Camp, last room on the table)
• Rannveig's Fast south of Morthal, northeast of Rorikstead (after falling down the trapdoor in fron of the chest and dragon-word-wall)
• Fellglow Keep northeast of Whiterun (straight up the stairs and to the right)
• Dainty Sload (A ship along the coast southwest of the Solitude Lighthouse)
• Sunderstone Gorge (Right of the alchemy on a table with skeletons)
• Solitude Palace (Queen's room)
• Yngvild, between Dawnstar and Winterhold (Very last room near the end)
• Treasury House in Markarth (End table in Thonar's bedroom, which is the left from the front door)
• Dwemer Museum in Markarth, entrance to the right as you enter Understone Keep (Table in room to the Left)
• Thalmor Embassy (Elenwen's Solar, second floor, first bedroom on your left) (Or in the cave through which you escape, after jumping from the ledge, on a small table beside a skill book and dead mage.)
• Hob's Fall Cave (Near the alchemy set on the left)
• Pinewatch - House Northwest of Helgen with a Bandit hideout underground. Gem is in a storeroom before the 'boss'.

Now there are also mods that place map markers for them or otherwise make collection easier:

• Stones of Barenziah Quest Markers
• Stackable Stones of Barenziah with Optional Quest Markers
• Sounds of Barenziah