[RPG] Would there be balance issues if I allowed opportunity attacks against any creature, not just hostile ones

dnd-5ehouse-rulesopportunity-attack

As shown in this question and answer, I am confused by the terms "hostile" and "enemy". So I was wondering if in my own game I could simply remove the "hostile" from the opportunity attack rules, because I think there it is particularly useless.

Currently, the rule on opportunity attacks says (PHB, p. 195; emphasis mine):

You can make an opportunity attack when a hostile creature that you can see moves out of your range.

I'm thinking attacks are allowed against allies; why shouldn't attacks of opportunity be?

Would there be balance issues if I house-ruled that opportunity attacks can be made against any creature, not just hostile ones? Are there any spells, effects or others that now unexpectedly work differently?

The revised rule would read:

You can make an opportunity attack when a creature that you can see moves out of your range.

Edit: Based on one of the answers about the warcaster, I can see why it shouldn't be like this. Now one of the reasons to allow one to attack an ally would be for example when you get turned against them, however they are not hostile towards you. So a solution should allow AoO against enemies that are not hostile towards you, but it shouldn't be possible if you don't actually mean to harm the enemy.

Best Answer

This Slightly Increases the Power of the Players

This is one of those rule changes that doesn't directly increase power level but instead increases the options available to the player. Fundamentally it should be balanced because NPCs and PCs can both use it, however it is the sort of thing players are far more likely be take advantage of than monsters.

A couple of examples where players could use this to their advantage:

War Caster Feat

As Patron Paton excellently points out, the War Caster feat has the benefit:

When a hostile creature's movement provokes an opportunity attack from you, you can use your reaction to cast a spell at the creature, rather than making an opportunity attack. The spell must have a casting time of 1 action and must target only that creature.

Your change would allow buff and healing spells to be cast on allies as they leave your reach. This significantly improves the action economy for support casters and opens up a lot of new combat combinations.

Raging Barbarians

AuxTaco points out a case where this rule could be used to keep your barbarian raging. The rage feature requires you to attack or take damage since your last turn or your rage ends. This modification would allow the barbarian to run past an ally and take damage thus continuing their rage, or alternatively the barbarian could attack an ally with their AoO to achieve the same result.

This is a somewhat meta-gamey option but it is a valid option per the rules with your modification. Perhaps situational and unlikely to make a major difference overall, but still a net positive for player options.

Other niche combinations

The two situations above are just examples of the kind of trickery you can pull off with this rule change, there are likely many more.

I can think of a strange combinations of a character who gains a benefit from reducing a creature to 0 HP and a death domain cleric who heals more effectively from 0 HP. Combining these abilities to knock down and then heal an ally could provide a net benefit to the party overall.

Only creatures you are hostile to

Based on some comments/edits you made and Admiral Jota's answer I understand you have recognised the issues with this rule and instead are considering a smaller modification.

You can make an opportunity attack when a creature you are hostile to that you can see moves out of your range.

The problem with this in my mind is that the players determine who they are hostile to and can easily exploit this. "I am hostile to the rogue, he stole my orange juice this morning". A good DM can easily deal with this but it would be better not to leave it open to interpretation.

Creatures you mean to harm

Your other "mean to harm" suggestion would have a rule something like this:

You can make an opportunity attack when a creature that you can see moves out of your range, provide you mean them harm.

This fixes some of the issues with the "that you are hostile to" variant but does open up some new ones. What counts as "mean them harm"? Some spells cast with War Caster deal no direct damage but are valid options.

Conclusion

Overall this house-rule isn't particularly unbalanced but does open up some interesting combinations. As most of these combinations actively reward intelligent play from the party I don't really consider that a bad thing. I suggest you allow this on a play-test basis and be prepare to modify/remove it should it cause issues.