There's no need to house rule this — you just need to start enforcing its restrictions. Allow me to draw your attention to its area of effect:

4 square″ + 1″ square/level

(Aside, this is taken from PHB page 95, since the Illusionist version of the spell has different range and area of effect.)

These are table inches, which we could convert to feet/yards, but the following will be simpler if we just compare table inches directly with fireball's numbers.

A fireball has a radius of 2″. Let us ask then, “what level would an illusionist have to be to cast a phantasmal force large enough to contain a 2″ radius sphere?” Let's make a first-order approximation using the minimum number of 1″ squares necessary to create an area in which the image of the sphere could appear. The area of effect of fireball is three-dimensional, but the area of effect of phantasmal force is two-dimensional, which presents an odd issue. However, phantasmal force is described as being able to create images “within the boundaries”, so let's be as generous as the spell seems to be intended, and say that the vertical space it can create the illusion within is “good enough” for our purposes and we just need the footprint of the illusory fireball to fit inside the phantasmal force's area of effect.

We'll assume as a first-order approximation that each 1 square″ of area of effect can't be divided, so they have to be tiled as whole square table inches. Inscribing a 4-unit sphere's footprint (a circle) inside a pixelized shape requires a 4×4 square of squares, or 16 tiled squares.

So, as a first approximation, an illusionist needs to have 16 square table-inches of area of effect to play with to be able to make a phantasm of a full fireball. That requires being level 12 to pull off.

But maybe you think tiled squares aren't generous enough. Let's allow the table-inch squares of area of effect of the phantasmal force to be subdivided infinitesimally so that the area can be a cylinder that just barely contains the footprint of an illusory fireball and see where that gets us. (Some DMs might reject this move since the area of effect is given in squares and they may require a straight-edged area of effect, but some others might allow it with the reasoning that it's fine as long as the final area matches regardless of shape.)

A 2-unit-radius sphere has an area through its widest point of just over 12½ units, rounding up to 13 square″ required. So even being super-generous, it would still take an illusionist of level 9 to use phantasmal force to create an illusory fireball.

Clearly, creating illusions of fireballs with phantasmal force isn't something that your players should be doing at first level, or even 5th level when their magic-user friends are starting to throw real fireballs.

Conclusion: You don't need to house rule, just give the players a reality check

So, there you have it: you don't need to house rule this at all, you just need to pay proper attention to the relatively small area of effect, and break the news to your players that you've been accidentally letting them create a 9^th-level illusionist's phantasmal forces before they were actually 9th level, and won't be doing that any longer.

If they are 9th- or 12th-level, then this isn't overpowered at all, because they have access to equally (or more) powerful effects already and cleverly using their resources like this should be letting them paste low-HD enemies by the truckload. So again, there simply isn't a problem if the area of effect is enforced.

Addendum: Don't overlook the audio limitation either

The above is all completely ignoring the impact of the stipulation that a phantasmal force is silent. Is a fireball silent? Not at all. An illusory fireball with no sound effects is not very convincing, and unlikely to fool even stupid opponents, even stupid opponents who have already been hit by a real fireball. In fact, having just experience a real one with all the sound and fury of real fire, they'd be more likely to notice that a silent one is not quite right.

Yes, the affected target can hear the other bug bear, but it might not matter because illogical outcomes are internally rationalized by the affected creature.

Source: PHB pg. 264, Phantasmal Force

The target rationalizes any illogical outcomes from interacting with the phantasm.

So if the other bug bear was screaming at him because he was being eaten, the one who was affected by the spell might make an Investigate check to discern the illusion. But if he failed, he may rationalize screaming food as:

1. Another bug bear trying to wrest away his food;
2. A commotion in a different room nearby;
3. His own internal thoughts pestering him for being so gluttonous;
4. etc.

However, a key bit here that's been missed is that the affected bug bear is not compelled to eat by the spell just because it is hungry. So it wouldn't necessarily attack a giant plate of food that's arguing with it. It may just end up arguing with the food. At this point, it's entirely in the DM's hands to roleplay this out.

It's certainly a creative use of the spell, but understand the illusion would simply make the other bugbear look like food. It wouldn't actually make the affected bug bear try to eat him, especially if the food ran away or started fighting back.