Conjure Dragon Priest spell

You are 100% correct in your misgivings. Due to destruction magic not scaling, you will lag behind in terms of damage done late game (especially if you up the difficulty). The only way you can make up for this is via the enchanting perks that lower destruction magic costs so you can cast and spam more spells. If you can get it down to 10% or less cost via multiple enchanted items, it is still viable due to the fact that you have instant stagger at your beck and call with the dual casting and impact perks, not to mention heavy aoe with fireball and such.

If you don't mind modding, there are already mods out there that improves destruction magic.

Finally, you'll still need a backup plan in case you run into some highly magic resistant foes (Breton with dragonskin activated, for instance.)

My mage knowledge is mostly second hand but I can give out these pointers so far:

• You can not get a higher tier perk with out getting the lower ones. For example you can do nothing in the destruction tree with out starting with the Novice perk. This applies for all of them as well.
• There is no end all be all elemental skill. A lot of people find Electrical the most useful simply cause it instantly strikes what is under your target, you do not have to lead your target. But Ice works against fire based dragons (and I've found it very useful against giants), fire works against both kinds of trolls, all of the undead, and electrical works good against mages. If you are trying to be an offensive mage you will really want to go with all of the elements in the long run so you are never stuck with just one choice.
• Impact is pretty much a Must Have. When my brother got this he went from having to do some fancy kiting of enemies to being able to constantly stumble them. This is -not- the win button, but it definitely puts you on a bit more even footing than you are with out it.
• When considering Conjuration keep in mind that the max level summons, known as thralls, are permanent until they die. You will need to have enough mana on hand to cast them, but it may not be worth while to put perks in here aside from those that allow you access to have 2 summons at once.

I hope this high level overview has been helpful and addressed your questions. Any more concerns ask in the comments and I will try and add in more information as I currently understand it.