How long does it take for plants to respawn

You are 100% correct in your misgivings. Due to destruction magic not scaling, you will lag behind in terms of damage done late game (especially if you up the difficulty). The only way you can make up for this is via the enchanting perks that lower destruction magic costs so you can cast and spam more spells. If you can get it down to 10% or less cost via multiple enchanted items, it is still viable due to the fact that you have instant stagger at your beck and call with the dual casting and impact perks, not to mention heavy aoe with fireball and such.

If you don't mind modding, there are already mods out there that improves destruction magic.

Finally, you'll still need a backup plan in case you run into some highly magic resistant foes (Breton with dragonskin activated, for instance.)

I believe once you have taken down a Giants Camp (and it is confirmed as cleared on the map) it takes around 2 weeks (ingame) for the Camp to respawn - I read this somewhere when I was looking over Giant Camps for the toes too. However, they still roam freely around the open plains with mammoths. Its just a case of running around for a while until one spawns.