It would be very unbalanced, especially in the long-run
I understand some of the logic behind this: some cantrips can critically-hit, and those would deal lots of damage when they do hit despite being cantrips. However, what the DM fails to realize is that a Rogue can also do this much damage or more with a Sneak Attack Critical, and on a hit-die (1D8 for the Rogue) that is equal-to or greater-than the three typical dedicated spellcasters: Wizards (1D6), Sorcerers (1D6), or Warlocks (1D8).
This means that a smart-enough Rogue can deal more damage on a critical, has more health than a dedicated spellcaster, and does so on a higher frequency, since, to get Sneak Attack Damage, they have to be rolling to hit on-Advantage (which is why some Rogues never go anywhere without an ally near them), which improves the likelihood of at least one of the dice rolling a Critical.
A Critical Hit doubles the number of dice rolled for damage, or, for some DMs who would rather simplify it, it doubles the amount of damage rolled on the standard amount of dice.
Let's compare:
- The Eldritch Blast Cantrip makes a Ranged Spell Attack and, when it hits, it does 1D10 Damage. The "average" roll for 1D10 is 5-6 points of damage. A Critical Hit would deal 2D10 points of damage, which averages to around 11. This will critically-hit at a 5% chance.
- A Rogue with a Light Crossbow makes a Ranged Sneak Attack, and, when it hits, it deals 1D8 points of damage. The Rogue rolls on Advantage, because they have to be doing so to get the Sneak Attack, and Rolling on Advantage roughly doubles your chance of critting, so, instead of having a 5% chance to roll a 20, the Rogue has ~10% chance to critically-hit. In-Addition, a First-Level Rogue rolls 1D6 in-addition to whatever damage they do, and, when you critically hit, those Sneak Attack Dice are doubled, too. So, that ends-up 2D8(9) + 2D6(7), or with an average damage of 16.
Saving Throw Spells
But there are other spells, yes. These force the opponent into performing a Saving Throw and can never critically succeed or critically fail. Many of them, especially at higher levels, still deal damage when the opponent succeeds their Saving Throw, but don't do as-much or don't cause additional effects.
In a sense, for a damage-dealing capability, the removal of the ability of Attack-Spells to critcally-hit makes them much-less preferable than Spells that cause Saving Throws, since Ranged Spell Attacks on-failure never do damage while most of the Saving-Throw spells are still useful when the opponent succeeds their DC.
But wait, your DM is allowing Critical Successes and Failures on Saving Throws! And this is where the balance really gets broken: Most of the spells that used Ranged Spell Attacks are designed to do moderate single-target damage, while the spells that cause Saving Throws either do a lot of damage and/or bestow a crippling effect on one target, or do a sizable amount of damage and/or a strong effect to multiple. Think of the Cleric's Spirit Guardians spell, which deals 3D8 damage on a failed save and half as much on a successful save. Adding critical successes and fails means that any opponent that enters the area or starts their turn there could take as little as 1/4th damage on a critially-successful save to as much as 6D8 damage on a critically-failed save.
Essentially, this ruling makes a great portion of the spells on the Spell lists much less useful, makes Saving Throw Spells much more preferable, and doesn't compensate the Spell-Attack spells for this reduction in viability.
Adamantine armor
It has the property:
While you're wearing it, any critical hit against you becomes a normal
hit. (DMG 150)
It is only classified as uncommon, but it is still a magic item. Depending on how the DM handles that, the difficulty of acquiring one may change. You also need to be proficient in at least medium armor to properly use one.
Note
Rolls with Disadvantage have a really low chance (1/400) to be crits. So if you impose it on an attack (eg. by dodging or the blur spell), you can improve your chances significantly.
Best Answer
Combat Would Go Slower
Because the damage being dealt is slightly lower than normal, combat would go slower with this type of house rule.
Which is great if that's something you wish to see happen - combat is a big draw for some players. But it does affect a few other game mechanics too...
Players Will Be Able To Tell Their Opponent's AC
It also means they'll know exactly how high a roll they need to make if they hit the monster's AC - which isn't a huge advantage in a fight, but something they might reasonably be able to figure out in actual combat (how hard it is to hit something, that is).
Higher Damage Characters/Monsters Will Have A Slight Disadvantage
Classes that are proficient in dealing high damage will have a slight disadvantage under these rules - they'll be dealing less damage on average and thus will be slightly less effective at their job.
Higher AC Characters/Monsters Will Have An Advantage
Characters and monsters with high AC will be taking slightly less damage on average, and thus won't go down quite as quickly - they'll be slightly better at their job.
Some Mechanics Will Be Affected
There are already some mechanics that half damage - off the top of my head, Barbarians take half damage to common damage sources when under Rage - so you might have to half the damage twice for a hit that hits AC - in addition to spells which might also do half-damage (and Barbarians who have the Bear Totem class feature) - which complicates things a bit, but not too severely.
Overall, This Isn't A Bad Change
Under this house rule, damage will be done slightly slower and battle will take slightly longer, but high AC characters will feel rewarded for their high AC, so it might be a net benefit.
It also might make some mechanics more difficult to manage, but not impossible and only in rare occasions would it come into play.
So overall it sounds like a good rule for making it feel a bit better for players who just barely missed having their armor deflect a blow, but might slow combat down a little.