Unlike some previous editions in which the fireball will adjust based on the space available and spread down corridors if contained, 5th ed simply states that the fireball will take up the amount of space listed as 20' radius sphere and spread around corners to fill that space, never expanding to exceed that distance from the point of origin. This specifically breaks the line of sight rule on page 204.
So if cast at the entrance of a 10x10 room with a 5' corridor leading into it, it shall fill that room and spread 20' down the corridor.
Fireball PHB PG 242
The fire spreads around corners.
D&D is not a combat simulator
All games make compromises between playability and simulation. Chess, for example, falls on the side of playability while the simulators the Air Force uses to train their pilots fall on the side of simulation. As war games go, D&D is simulation light, playability heavy - tending more towards Chess than an F-16 simulator.
I appreciate that you understand that a creature does not occupy a whole 5-foot square, however, it goes further than this: combat does not actually happen in a series of discrete turns. Its funny to contemplate the fighter, just as the orc raises his scimitar for a killing blow, holding up his hand and saying "Wait, its the wizard's turn". Further, even though a creature is placed inside a 5-foot box for playability reasons its perfectly feasible that over the course of a round they are darting about all over the place in and around that box in order to get the clear shot they need.
As another complication, many spells do not indicate that something physical travels from the caster to the target - are these to be treated differently in your system? Notwithstanding, for those that do, there is no reason to suppose that a magical effect needs to follow physical laws - I can easily imagine the green ray from Disintegrate passing through the bodies of several creatures without harming them on its way to the target. Alternatively, who says that these effects go in a straight line? Maybe they zig-zag around the intervening creatures because of, say, magic?
Physical weapons do not go in straight lines either: they follow a ballistic trajectory. Arrows "arch" which is why we have the word archery - historically missile troops shot over the heads of their melee companions to drop death from above on the enemy - much like modern artillery and air strikes do today.
I judge that this means see clearly, i.e. for long enough to cast the spell.
I disagree, seen is seen but even if I did accept your judgement casting most spells takes 1 action. How long is that, exactly? As long as it takes to swing a sword? Clearly not, because a high-level fighter can swing his sword multiple times in the same time period and move in between. The rules are clear that a round is about 6 seconds, however, there is no RAW for how long a turn or an action is.
However, if you have considered all of the above and still find that it grates on you then you might consider a house rule. These are only worthwhile when they add more to the game than they cost.
Personally, I don't feel the rule adds anything much but that is a judgement call and your judgement is not mine. Here are what I see the costs are:
- It will rarely come into play. The best combats in D&D 5e pit approximately equal numbers against one another: 4 on 4 or 5 on 5 given a typical party. With so few combatants on the grid there is almost always going to be a spot within movement that a ranged attacker can get to so that they have a clear path to the target.
- It will slow down combat. Each ranged character, on both sides, will spend a little bit longer each turn positioning themselves just so to avoid these penalties. Now, I play Advanced Squad Leader and I can happily spend hours working out the exactly right hex to place my anti-tank gun in or minutes considering if I move this squad across that road just here will his MG nest be able to see them (and more to the point, kill them) but that is the game experience I want from ASL - I don't want it from D&D.
- Its verbose. I realize that this is a draft of the rule but it runs to 6 paragraphs and has a largish number of sub-cases and wherefores that the players have to keep track of - all while remembering which spells are affected and which aren't.
This is what I do:
- I use the RAW but I "eyeball" it - if the creature concerned can easily get somewhere (5-10 feet away) where they have a clear path I just assume they do that and don't get the modifier. I just take it as a given that the creature takes their shot at the most advantageous time.
- If the players decide to engage in combat in e.g. a crowded marketplace full of civilians then I just say "No ranged attacks or large AoE, there are too many innocents about". Of course, their enemies might not be so squeamish but that's part of the fun of being a DM.
Best Answer
Boy, so many people lining up to tell you "don't do it that way it's badwrongfun!" I'll offer a differing perspective, which is yes, absolutely, use a house rule to this effect. It has the desired effect of adding verisimilitude without "nerfing" or "ruining" anything. I shall offer up real play experience and not pure opinion to demonstrate this.
I used this exact kind of house rule during all of my AD&D 2e days (a decade) and from time to time in 3e/Pathfinder days for the same reason; adding some verisimilitude to combat - far from negating the need for tactics, it instead makes you have slightly better tactics because you aren't relying on things being exact (like artillery and infantry in the real world). Planning for the possibility of friendly fire makes for incorporation of real world techniques which is always desirable to me (learning about real world weapons, tactics, history, science, etc. is one of the best benefits of RPGs that many people seem to want to stomp out nowadays). Also, adding a little bit of randomness to magic makes it not so overwhelmingly better than the martial options - some of the martial/magic power differential comes from "I have to roll all the time" vs "I just do it", so by making your mages roll to do things you equalize the playing field a bit.
House Rule Option 1: Grenade-Like Missiles
But to do this you need something that best as I can tell is missing in 5e, which is what previous editions call "grenade-like missile," "scatter diagram," or the "throw splash weapon." In 1e AD&D it was under Grenade-Like Missiles in the DMG (p.64) - you'd roll d8 for direction and d4 (short range) d6 (mid range) d8 (long range) for how far it landed from the target.
In its most recent 3.5e incarnation, it's written like this:
In other words, roll to hit the target place, and use d8 to scatter direction on a miss.
So, super helpful for those thrown mundane firebombs, but you can also use it for area effect spells that don't usually require a hit roll - you just require that touch attack on their target or target square. (You have to interpret what "range increment" means for the spell, I usually just did 1 square if it was dropping in short spell range, 2 squares medium, 3 squares long). You can use a real ranged hit roll or sub in something like an Arcana check at your discretion. (Sometimes in 3e I've used Spellcraft instead of a to-hit).
A fireball is most likely still going to hit that main guy you throw it at. It is less likely to perfectly get every one of a group of enemies, and it's also more dangerous to try to crisp everyone including the enemies in direct melee with your party members.
Then you get to apply other bonuses/penalties like you would for ranged attacks. I'm running a Pathfinder pirate game where PCs are often trying to heave fireballs and lightning boats at enemy ships while going full speed in a ship heaving up and down on the waves through a rainstorm. Those additional to-hit penalties make it really exciting, especially if some of their crew has already flown over onto the enemy ship! Similarly, if players play smart and set out ranging stakes and practice, then when the hordes of wild elves attack their encampment they could get advantage on the placement roll.
House Rule Option 2: Proximity Saves
One of my current Pathfinder GMs handles this same problem just by making all party members in close combat with anyone getting AoEd make a save for half/no damage, for the same reason. This works OK but I don't like it as much, mainly because it takes all the control away from the wizard. "Having to roll to hit" is not onerous and expected of other classes, but just saying that they can't really try to place it better and you just have to save if you're around is a bit odd and tends to just make AoEs one radius bigger. "How come an enemy in contact doesn't have to save too?" Although rogues and monks and such enjoy being able to use evasion with impunity in these situations.
Why This Isn't A Huge Mistake
The theoretical problems levied against house ruling this by other posters are invalid.
"This takes control away from the player!" Only to the degree that control is taken away from any character (especially the fighty types) from having to roll a die to determine success. That's arbitrary, and other d20 derived games do have spellcasters roll for success and it doesn't "ruin" them.
"This obviates the point of using a grid with its numerological perfection." No it doesn't, again, any more than fire bombs or ballista or anything else having to roll to place on a grid does. You don't have to go Theater of the Mind to get a little bit of imprecision. You could, but that's not a necessary solution.
"This removes the need for tactics!" Really? Removing randomness removes tactics? Counterexample: the real world. Rather than removing tactics this instead ensures more realistic tactics.
All kinds of tactical tips for your monsters to not bunch up and not get all fireballed. All fine, do those too, but does not bear on the validity of this house rule. Encounter design is an entirely different topic.
"It's not RAW." No one cares; your game is your rules. See How do you help players not focus on the rules?
"This must mean you are a meanie and are against your players." I assume you'll use the same rule when monsters fireball, so that's not really valid, and it assumes bad faith in that you don't really just want more realism like you say but you want to "stick it to mages." Plus, does it mean you hate fighters because you make them roll for stuff?
From my experience, all this house rule does (both variants) is make a) casters not drop AoEs sometimes in risky situations, and b) make martials have to be a little smarter because they can't rely on pinpoint spellcaster artillery precision. It changed the game a little, but not in any way any of the participants thought was "unbalanced." Instead, it ended up more realistic, which is one of my core gaming principles (YMMV). I haven't used these in 5e, but I am familiar enough with 5e to say I don't see any way in that this would have a different effect (as to be expected, as 5e is closest to a 2.5e in practice).