There's no need to house rule this — you just need to start enforcing its restrictions. Allow me to draw your attention to its area of effect:

4 square″ + 1″ square/level

(Aside, this is taken from PHB page 95, since the Illusionist version of the spell has different range and area of effect.)

These are table inches, which we could convert to feet/yards, but the following will be simpler if we just compare table inches directly with fireball's numbers.

A fireball has a radius of 2″. Let us ask then, “what level would an illusionist have to be to cast a phantasmal force large enough to contain a 2″ radius sphere?” Let's make a first-order approximation using the minimum number of 1″ squares necessary to create an area in which the image of the sphere could appear. The area of effect of fireball is three-dimensional, but the area of effect of phantasmal force is two-dimensional, which presents an odd issue. However, phantasmal force is described as being able to create images “within the boundaries”, so let's be as generous as the spell seems to be intended, and say that the vertical space it can create the illusion within is “good enough” for our purposes and we just need the footprint of the illusory fireball to fit inside the phantasmal force's area of effect.

We'll assume as a first-order approximation that each 1 square″ of area of effect can't be divided, so they have to be tiled as whole square table inches. Inscribing a 4-unit sphere's footprint (a circle) inside a pixelized shape requires a 4×4 square of squares, or 16 tiled squares.

So, as a first approximation, an illusionist needs to have 16 square table-inches of area of effect to play with to be able to make a phantasm of a full fireball. That requires being level 12 to pull off.

But maybe you think tiled squares aren't generous enough. Let's allow the table-inch squares of area of effect of the phantasmal force to be subdivided infinitesimally so that the area can be a cylinder that just barely contains the footprint of an illusory fireball and see where that gets us. (Some DMs might reject this move since the area of effect is given in squares and they may require a straight-edged area of effect, but some others might allow it with the reasoning that it's fine as long as the final area matches regardless of shape.)

A 2-unit-radius sphere has an area through its widest point of just over 12½ units, rounding up to 13 square″ required. So even being super-generous, it would still take an illusionist of level 9 to use phantasmal force to create an illusory fireball.

Clearly, creating illusions of fireballs with phantasmal force isn't something that your players should be doing at first level, or even 5th level when their magic-user friends are starting to throw real fireballs.

Conclusion: You don't need to house rule, just give the players a reality check

So, there you have it: you don't need to house rule this at all, you just need to pay proper attention to the relatively small area of effect, and break the news to your players that you've been accidentally letting them create a 9^th-level illusionist's phantasmal forces before they were actually 9th level, and won't be doing that any longer.

If they are 9th- or 12th-level, then this isn't overpowered at all, because they have access to equally (or more) powerful effects already and cleverly using their resources like this should be letting them paste low-HD enemies by the truckload. So again, there simply isn't a problem if the area of effect is enforced.

Addendum: Don't overlook the audio limitation either

The above is all completely ignoring the impact of the stipulation that a phantasmal force is silent. Is a fireball silent? Not at all. An illusory fireball with no sound effects is not very convincing, and unlikely to fool even stupid opponents, even stupid opponents who have already been hit by a real fireball. In fact, having just experience a real one with all the sound and fury of real fire, they'd be more likely to notice that a silent one is not quite right.

(Disclaimer: the power of the illusions are controlled by the DM unlike most spells, the reason is because illusions might be too powerful and game changing for so little. Therefore, the best course of action is to ask the DM about your question, if you are the DM here are some guidelines but at the end is up to you)

Narrative wise

From the DM to the players, the way they present the illusion is the way it works (mostly, rules of the spell still apply). This is very important, specially in how the magic works in the world, for example, shield: does it make sound when you cast it or while it is active? What about light or shimmers? Some illusions like silent image and minor illusion+ don't make sound. A beast probably won't question why the wall that suddenly appear didn't make sound, but an intelligent creature familiar with a tiny bit of magic, yes, it would.

An interesting spell, narrative wise, is major image. Since major image can create illusion with smell, temperature, and sound, etc, the narrative of the spell should reflect the illusion presented. A wall emerging from the deeps of the earth making crumble sounds and the characteristic smell of earth, dust and cut grass will be way harder to disbelieve than a wall that just appear.

It just odd

This is the most basic way of disbelieving an illusion. Something might be misplaced and should not be there. For example, a cardboard box in the middle of a room undoubtedly will attract attention but the same box in a corner won't be noticed as much. Another example would be that the caster was too greedy and created an illusion so big that there wasn't room for a fake shadow. A giant guppy floating in the middle of the battlefield is a tell too. An example list would be:

• The illusion is in a place where it attracts attention
• The action used by the illusory creature is out of its common behavior
• The spell does not make sense (the wall just pop in from thin air with any "common" warning, like displace air current and so on)
• The creature do nothing at all relevant, e.g. it does not attack.

Failed roll

Some DM will ask for roll checks when you want to create an illusion of a creature that you have not seen or that you usually do not use. These checks vary from illusion to illusion, the most common are arcane, nature and religion. A failed roll just increase the oddity of the illusion, it does not make an automatic failure, the "enemy" still have to pass the corresponding check if the illusion seems odd for him. For example, you know of a giant floating fish, you have seen one once, but you fail (nature roll) to recall the details, but the Orc that you are facing only heard of tales of such creature, he have to pass an history check to remember details of such creature. On a failed, the big fish is safe. This goes for hallucinatory terrain and other spells that require some degree of knowledge.

Phantasmal Force

This is a special case, mainly because of two particular things:

1. While a target is affected by the spell, the target treats the phantasm as if it were real.
2. The target rationalizes any illogical outcomes from interacting with the phantasm.

If we consider the initial save roll as failed, there is very little that the affected target can do for itself. Number 1 take care of things like thinking it is to odd for it to exist or to be real. Number 2 is particular problematic, since it make any interaction feel real and in order no matter how illogical the reaction or action was. But, there is a solution to this problem, third parties. Since any third party won't see what the affected creature sees, it is very easy for them to connect the dots and establish that something is very wrong with him. They can yell it's in your head or it's an illusion, they can ask what is wrong and latter say there is nothing there or the like. In that case, the affected target have a reason to suspect that it might not be real and act accordingly.