Told to leave in lakeview Manor

Avoiding the bug

About the Forsworn Conspiracy quest:

Followers may get permanently stuck in the Shrine of Talos, regardless of if the quest is completed and you escaped from Cidhna Mine. If this happens, the follower will not leave even after you tell them to go home, and if they are moved elsewhere via console, they will walk back to the Shrine of Talos, no matter where in Skyrim they are.

^{From: UESP Wiki, "The Forsworn Conspiracy - Bugs"}
I'd recommend not having a follower while doing this quest to avoid them getting stuck in the Shrine of Talos. Or you could ask them to wait outside the Shrine. Multiple backup saves are also recommended.

Fixing the bug

If you've already encountered the bug and can't load a previous save, try these*:

Try the enable and disable console commands.

Make a save game first, and then in the game, open up the console by pressing ~, then click on your follower (in OP's case, Aela). Aela's RefID (0001A697) should show up in the screen. If the RefID is correct (meaning you have selected Aela correctly, and not another object), type disable and then enable in the console. This should 'reset' her.

If Aela is not near your location, you can still select her by entering the console command prid 1A697, after that type disable and then enable. Then try the moveto player command (which moves that prid-targetted entity near the vicinity of the player) and see if she gets 'reset' and decides to stop her pilgrimage to the Shrine of Talos.

If it still doesn't work, try finishing the Forsworn Conspiracy quest and the next quest that it will trigger, No One Escapes Cidhna Mine. See if your follower will now follow you.

^{(I know OP has already posted a solution that worked for them. Just posting alternative solutions in case someone else experiences a similar issue.)}

It's because when you built the house it didn't add to the number of houses you own. Meaning that, as far as the game goes, the house doesn't even exist. It is a bug. A very, very, VERY, ANNOYING bug. -__- There is no fix for it. You will have to load back before you built or purchased the land, until it shows in your game stats that you own another house. (Viewed from the pause menu) That is, if you see it that important for your spouse and kids to live at Lakeview Manor...