There's no need to house rule this — you just need to start enforcing its restrictions. Allow me to draw your attention to its area of effect:
4 square″ + 1″ square/level
(Aside, this is taken from PHB page 95, since the Illusionist version of the spell has different range and area of effect.)
These are table inches, which we could convert to feet/yards, but the following will be simpler if we just compare table inches directly with fireball's numbers.
A fireball has a radius of 2″. Let us ask then, “what level would an illusionist have to be to cast a phantasmal force large enough to contain a 2″ radius sphere?” Let's make a first-order approximation using the minimum number of 1″ squares necessary to create an area in which the image of the sphere could appear. The area of effect of fireball is three-dimensional, but the area of effect of phantasmal force is two-dimensional, which presents an odd issue. However, phantasmal force is described as being able to create images “within the boundaries”, so let's be as generous as the spell seems to be intended, and say that the vertical space it can create the illusion within is “good enough” for our purposes and we just need the footprint of the illusory fireball to fit inside the phantasmal force's area of effect.
We'll assume as a first-order approximation that each 1 square″ of area of effect can't be divided, so they have to be tiled as whole square table inches. Inscribing a 4-unit sphere's footprint (a circle) inside a pixelized shape requires a 4×4 square of squares, or 16 tiled squares.
So, as a first approximation, an illusionist needs to have 16 square table-inches of area of effect to play with to be able to make a phantasm of a full fireball. That requires being level 12 to pull off.
But maybe you think tiled squares aren't generous enough. Let's allow the table-inch squares of area of effect of the phantasmal force to be subdivided infinitesimally so that the area can be a cylinder that just barely contains the footprint of an illusory fireball and see where that gets us. (Some DMs might reject this move since the area of effect is given in squares and they may require a straight-edged area of effect, but some others might allow it with the reasoning that it's fine as long as the final area matches regardless of shape.)
A 2-unit-radius sphere has an area through its widest point of just over 12½ units, rounding up to 13 square″ required. So even being super-generous, it would still take an illusionist of level 9 to use phantasmal force to create an illusory fireball.
Clearly, creating illusions of fireballs with phantasmal force isn't something that your players should be doing at first level, or even 5th level when their magic-user friends are starting to throw real fireballs.
Conclusion: You don't need to house rule, just give the players a reality check
So, there you have it: you don't need to house rule this at all, you just need to pay proper attention to the relatively small area of effect, and break the news to your players that you've been accidentally letting them create a 9th-level illusionist's phantasmal forces before they were actually 9th level, and won't be doing that any longer.
If they are 9th- or 12th-level, then this isn't overpowered at all, because they have access to equally (or more) powerful effects already and cleverly using their resources like this should be letting them paste low-HD enemies by the truckload. So again, there simply isn't a problem if the area of effect is enforced.
Addendum: Don't overlook the audio limitation either
The above is all completely ignoring the impact of the stipulation that a phantasmal force is silent. Is a fireball silent? Not at all. An illusory fireball with no sound effects is not very convincing, and unlikely to fool even stupid opponents, even stupid opponents who have already been hit by a real fireball. In fact, having just experience a real one with all the sound and fury of real fire, they'd be more likely to notice that a silent one is not quite right.
Best Answer
There is more work for you to adjudicate which effects will cause damage. It would make destroying objects a lot easier. Additionally some effects with saving throws don't make sense for objects. Finally it may be unfun for your players.
Extra work for you
The game has been balanced around the assumption that objects are hard to destroy unless they are explictly targeted. It makes the game simpler.
The effects that damage objects are intentionally few and far between. For example Shatter specifically damages objects, while the spell Blight does not (unless the object is a plant). Because of this you will need to consider which damage types will damage objects and which won't.
For example:
You will need to track which objects are in range of the spell, what they are made of, and what vulnerabilities and resistances each object has. Taking all of this together means you need to be more meticulous when you map out individual environments, with exact placements of objects.
Easier to destroy objects
If all AoE spells damage objects, then they will also damage buildings. Damaging buildings is something spellcasters in particular will want to avoid (espescially if they are in them). As a result you would expect magic users to want to limit the damage they do to structures.
Any self respecting adventurer wants to maximise loot gained and minimise the effort to get that loot. AoE spells satisfy both of those criteria, but only if they don't damage the loot. By making these types of spells damage objects you are changing the motivations of the generations of spellcaster who went before your players.
Unfun for your players
Changing their AoE spells to cause damage to objects will mean your players end up damaging their loot. The two potential effects of this are:
Neither of these outcomes will necessarily be fun for your players (or at least for most players). Battles with lots of enemies will be harder and/or the players will get rewarded less for having those battles.
Does all of this mean you shouldn't do it?
Not necessarily, but I would recommend talking to your players and seeing if they will enjoy the change.
Ultimately when we play D&D we are playing a game, not a reality simulator. The aim of games is to make sure everyone playing has fun (including the DM). If a homebrew rule change is implemented that potentially could make the game less fun for the players, that change needs to be discussed.
The flip side of this question is, of course, does the suspension of disbelief caused by not having this change make it less fun for you as the DM?
Only you can answer this question, but it should form part of the discussion you have with your players.
Does the game have rules for improvising this sort of damage?
Yes. Chapter 8 of the gives a table which sets out different levels of improvised damage, with suggested examples. Using this table, along with the damage output by the spell or AoE effect, is the place to start.