Normally no. But in the right circumstances, yes.
In most cases, a creature is assumed to be moving around in its space on the battlefield, not fixedly focused forward on one opponent (that's suicidal, for exactly the reason you present). So, normally, no, you can't just walk around an opponent and get advantage on them: they see you, see where you went, and are keeping an eye on you.
The exceptions happen in, well, exceptional circumstances, as judged by the DM's good sense. To demonstrate, I have to make something up, so let's have an example.
You meet an man in the forest. He challenges you, drawing his sword.
Suddenly your warlock companion throws a curse at the him, acting quicker than the drawn sword. You've seen this before, it's an eyebite curse. The man collapses, asleep.
“What was that for?!”
“We're looking for those old ruins, right? He might know where they are. No sense killing him out of hand just because he's jumpy. Get around behind him and be ready to hit him if he causes trouble.”
You stand a close behind where the man fell as the warlock shakes the man awake, and gets right up in his face, holding his attention. “That was impolite of you. Shall we start over?”
There, in that moment, you're not hidden. You didn't have to make a hide check to get there, but notice how you're out of the target's sight. This is an example of not being seen.
DM's judgement
The DM's judgement is very important in 5e, for exactly this reason. Most of the time it will be obvious: you're not hidden, and the target can see you just fine. In rare circumstances — often, circumstances that you've put some work into creating — you'll be unseen despite not being properly hidden, and then you'll have advantage if you need to put a knife in their kidney without actually hiding.
Assumptions
I'm going to be making the following assumptions, based on what you've already provided:
- 3rd level (since you get a crit on a 19 or 20)
- 16 Strength (no ASI to bump up to 18)
- Fighting a CR 3 creature (for base math)
- Average damage is 7.50 (4.50 from the die +3 Str mod)
- Average crit damage is 12.00 (4.50 per die +3 Str mod)
- Attack bonus is +5 (+2 prof, +3 Str mod)
- The enemy has AC 13 (per the DMG guidelines on page 274)
- DPR calculations are:
- Crit damage = Crit % x average crit hit damage
- Normal damage = Hit % x average hit damage
- hit % = 100 – [miss %] – [crit %]
Note that any flat modifier to the damage total won't change with a crit, since you only double the dice rolled, not the modifiers added.
Champion Fighter
Per the DMG page 274, a CR3 creature has an average AC 13, meaning you need to roll an 8 or higher.
With Advantage
AnyDice can tell us our miss chance and our crit chance. From there, we know our hit chance.
- A normal roll of 1d20 will have a 35% miss, 10.00% crit (1.20 DPR), and 55.00% normal hit (4.13 DPR) for a total DPR of 5.33
- A roll with advantage will have a 12.25% miss, 19.00% crit (2.28 DPR), and 68.75% normal hit (5.16 DPR) for a total DPR of 7.44
- A roll with your DMs +2 rule will have a 25.00% miss, 10.00% crit (1.20 DPR), and a 65.00% normal hit (4.88 DPR) for a total DPR of 6.08
With Disadvantage
AnyDice can tell us our miss chance and our crit chance. From there, we know our hit chance.
- A normal roll of 1d20 will have a 35% miss, 10.00% crit (1.20 DPR), and 55.00% normal hit (4.13 DPR) for a total DPR of 5.33 (unchanged)
- A roll with disadvantage will have a 42.25% miss, 1.00% crit (0.12 DPR), and 56.75% normal hit (4.26 DPR) for a total DPR of 4.38
- A roll with your DMs -2 rule will have a 45.00% miss, 10.00% crit (1.20 DPR), and a 45.00% normal hit (3.38 DPR) for a total DPR of 4.58
Rogue
How does this change affect other classes, specifically those who rely on bonus damage dice? I'm using a rogue for this example since sneak attack is easy enough to calculate, but a paladin falls under the same heading with their smite spells and the like.
We use the same percentages and base damage (assume Dex and a rapier) for our fighter, but we add sneak attack damage. That's 2d6 at level 3, so with advantage we add +2d6 (7) on a hit and +4d6 (14) on a crit. Attacks without advantage don't get sneak attack damage added in, so the disadvantage numbers from above carry over (I know you can get sneak attack damage without advantage, but we'll ignore that for simplicity).
With Advantage
- A normal roll of 1d20 will have a 35% miss, 5.00% crit (1.30 DPR), and 60.00% normal hit (8.70 DPR) for a total DPR of 10
- A roll with advantage will have a 12.25% miss, 9.75% crit (2.54 DPR), and 78.00% normal hit (11.31 DPR) for a total DPR of 13.85
- A roll with your DMs +2 rule will have a 25.00% miss, 5.00% crit (1.30 DPR), and a 70.00% normal hit (10.15 DPR) for a total DPR of 11.45
With Disadvantage
- A normal roll of 1d20 will have a 35% miss, 5.00% crit (1.30 DPR), and 60.00% normal hit (8.70 DPR) for a total DPR of 10
- A roll with disadvantage will have a 42.25% miss, 1.00% crit (0.12 DPR), and 56.75% normal hit (4.26 DPR) for a total DPR of 4.38 (identical to the fighter, as no advantage means no sneak attack)
- A roll with your DMs -2 rule will have a 45.00% miss, 5.00% crit (1.20 DPR), and a 50.00% normal hit (2.25 DPR) for a total DPR of 3.45
Conclusion
With your DMs proposed houserule, the expected DPR for any class is going to be decreased because of the fact that you're still only rolling 1 die, so the chance of a critical hit will not change. The biggest, well, advantage of rolling with advantage is it almost doubles your chance of a crit: 9.75% vs. 5.00% for a normal 20 crit and 19.00% vs. 10% for a champion fighter crit.
Indeed, that simple change reduces the overall expected damage output of the entire party, especially those classes that rely on burst damage in the form of more dice. As you gain in levels and get the extra attack feature, magic items/spells that add damage dice, and class features that change the damage dice done, the gap will only increase.
Best Answer
The asker mentions he is using a house rule:
From that, it's hard to say anything about how this rule works. We have had a DM ruling something along these lines in another question. As I mention there, this is a huge nerf for spellcasters in general. About your question:
There are no (default) rules for advantage for attacking from behind 1
One option we have is an optional flanking rule, which still requires another enemy close to you in order to actually give advantage. You can read more about it here. D&D usually assumes that the character has a 360 degrees field of vision, even if they are facing a specific direction in the grid, for the purposes of being attacked. There are also optional Facing Rules (DMG p. 252), which states
It is still unclear about an attack from behind coming from a PC that is in the front arc, as it's not a situation that would be usually happening. This is not comparable to casting a spell with an actual attack roll because then the spell would be coming from the same direction as the PC (i.e., front).
So, in general...
Your DM is making a lot of house rules.
Is that a problem? Up to your group. If you, or anyone else, is feeling that these house rules are making the game less fun, unfair or similar, talk to your DM and state your concerns. From my experience, new DMs sometimes make house rules without fully understanding the mechanical consequences of these house rules. Explaining that his modified rules are hurting your fun is usually the best way to make them see it.
You also mentioned your group is new to D&D 5e, while it's not clear if that's true for the DM as well, it is possible that he is not aware that he is using house rules. If that seems a possibility for you, check with him if he knows these rules are not official. Sometimes, simply stating "Hey bro, that's not how it works" is enough for the DM to notice "Whoops, I was thinking about other random system". I myself carried misconceptions from 3.5e to 5e when it was released.
Sadly, as these are house rules, we can't answer, from the rules perspective, more than "none of these rules is stated in the books" and "your DM is the one who makes the rules, so if he said it, he's kinda right" (but he should be consistent - if your spellcasters cast a spell on the enemies' backs, you should get advantage as well). You can make a different question asking whether or not these house rules are balanced and how they impact the game overall, but, as I said, this is a different question.
As a side note, although the Shatter spell is mentioned, the actual problem seems to be about the house rule. Shatter itself could have been a fireball or any other AoE spell.
As a second side note, the Facing also states
While "moves" probably means actual movement (walking from a square to another), if the DM is applying the same rule for AoE spells cast in a point of origin behind, he probably should allow you to react and change the direction you are facing to your back, when you see the spellcaster moving his hands and speaking some awkward phrases.
1 Note, however, that a specific rule is not required for Advantage/Disadvantage, as PHB p. 173 states
This is more a fall-back rule and a reminder that "the DM is in right to rule anything as he wants, to be frank", though, from how I read it.