Divine Smite says (emphasis mine):
Starting at 2nd level, when you hit a creature with a melee weapon attack, you can expend one spell slot to deal radiant damage to the target, in addition to the weapon's damage.
This wording is a little ambiguous as to whether Smite is part of the attack damage or a damage modifier. Damage modifiers aren't doubled. However, Sage Advice says that Smite is part of the attack damage and crittable since it does not require a save or its own attack roll.
The rest looks correct so you'd have:
Normal - 1d12 Slashing
- add Brute (MM p346) - 2d12 Slashing
- add Orcish Fury (Xanathar's p76) - 3d12 Slashing
- add Maneuvering Attack (PHB p74) - 3d12+1d8 Slashing
- add Divine Smite (PHB p85) - 3d12+1d8 Slashing + 3d8 Radiant
- add Crit, and Savage Attack (PHB p41) - 7d12+2d8 Slashing + 6d8 Radiant
- add strength bonus and Great Weapon Master (PHB p167) - 7d12+2d8+15 Slashing + 6d8 Radiant
Great Weapon Fighting (PHB p72) allows rerolls for an initial 1 or 2 on the d12 dice. The Radiant Damage and Superiority Die is in addition to weapon damage thus not covered by the GWF feature. (Sage Advice)
Notes:
- It's important to track the damage types due to possible damage resistance/immunity/reduction
- Brute (MM p.346) and Orcish Fury (Xanathar's p.76) are not core Half-orc feats; not all DM's will allow them.
Combat Would Go Slower
Because the damage being dealt is slightly lower than normal, combat would go slower with this type of house rule.
Which is great if that's something you wish to see happen - combat is a big draw for some players. But it does affect a few other game mechanics too...
Players Will Be Able To Tell Their Opponent's AC
It also means they'll know exactly how high a roll they need to make if they hit the monster's AC - which isn't a huge advantage in a fight, but something they might reasonably be able to figure out in actual combat (how hard it is to hit something, that is).
Higher Damage Characters/Monsters Will Have A Slight Disadvantage
Classes that are proficient in dealing high damage will have a slight disadvantage under these rules - they'll be dealing less damage on average and thus will be slightly less effective at their job.
Higher AC Characters/Monsters Will Have An Advantage
Characters and monsters with high AC will be taking slightly less damage on average, and thus won't go down quite as quickly - they'll be slightly better at their job.
Some Mechanics Will Be Affected
There are already some mechanics that half damage - off the top of my head, Barbarians take half damage to common damage sources when under Rage - so you might have to half the damage twice for a hit that hits AC - in addition to spells which might also do half-damage (and Barbarians who have the Bear Totem class feature) - which complicates things a bit, but not too severely.
Overall, This Isn't A Bad Change
Under this house rule, damage will be done slightly slower and battle will take slightly longer, but high AC characters will feel rewarded for their high AC, so it might be a net benefit.
It also might make some mechanics more difficult to manage, but not impossible and only in rare occasions would it come into play.
So overall it sounds like a good rule for making it feel a bit better for players who just barely missed having their armor deflect a blow, but might slow combat down a little.
Best Answer
Is it balanced?
Sure, it's balanced if you apply it to all sides of a combat equally. It's a very minor change. Even taking into account characters with multiple attacks each round, it's a couple of HP here or there - not worth worrying about.
Is it worthwhile?
No. You're adding complexity in the pursuit of verisimilitude. That's a fine idea, except that your method goes against the basic system design - D&D5E is not a margins-of-success system, it's a pass-or-fail system. There are other systems where the core mechanics include margin-of-success and have your glancing blows built in. That said, I don't think it adds enough verisimilitude to make the complexity worth while - you're adding an additional parameter check for a whole three damage.
D&D is not a reality simulator.
What you should be looking to do is describe your combats differently. Look at some real sword fighting if you can, or some Hollywood sword fighting if you can't. A sword fighter doesn't swing once every six seconds.
In D&D, one attack roll represents a series of swings, backpedals, feints, dodges, and the glancing blows you're looking for. It's all aggregated into a single roll to keep complexity down. Even "Extra Attack" isn't really extra attacks, the character isn't swinging significantly faster than they were before, but they're so good they can get in more blows that matter.
In the PHB (page 196) we have:
In other words, it's possible to lose hit points without taking any penetrating injury, from a glancing blow. Take a character with a shield: sometimes, the "hit" that "gets through" is not actual wounds, they're an expenditure of energy and effort in getting the shield in the way or the pain and discomfort from the enemy wailing on the shield. In a similar vein, a "hit" on a lightly armored character could actually be a "miss" dramatically, but the character twists or over extends something in the effort to get out of the way. It may not even hurt right now (adrenaline and all that), but it'll hurt later.