[RPG] How to decide when a paladin has fallen from grace in DnD 5e

You're right. There is no clear cut way. That's by design. How this works varies a lot by playstyle:

Some groups have a solid understanding of ethics and jurisprudence and use some limited form of said understanding adjudicate alignment in D&D. Others use their full understanding of ethics, which cumulates for them into a D&D compatible system, to perform that adjudication. These groups are very similar except that the first group tends to have less issues adjudicating alignment in difficult situations where the latter group discovers holes in their beliefs about morality. In both of these cases alignment is adjudicated according to the chosen moral/ethical system, and what constitutes oath-breaking will be adjudicated similarly according to the related principles of jurisprudence. As moral relativism becomes, unfortunately, increasingly popular, objective alignment becomes less and less so, especially amongst simulationist groups, and so these two methods of alignment adjudication are unlikely.

Some groups think alignment is stupid, meaningless, and entirely subjective. These people are very vocal. They likely also think that oathbreaking is similarly problematic. Like alignment-based powers in previous editions, in these groups if you're going to use anything that's based off alignment or alignment-like systems (like the paladin class) you should first check with your DM to find out 1) if it's allowed and 2) what it means in the context of the campaign. One common solution in the past in these kinds of groups has been just to waive the alignment restriction entirely, for the Paladin class.

Some groups think that obviously alignment is objective, because it's in the rules, but they just don't understand it yet. These groups will post difficult alignment questions as they come up on online fora. The outcome of alignment based actions in such groups will be fickle and likely a point of tension.

Some groups try and adjudicate this according to the original inspiration for the alignment system, which is a set of novels written by Michael Moorcock. This is hard because the novels are novel-y and don't explain exactly what Law and Chaos are in cut and dry terms. Such groups would probably not adjudicate the oath the same way they do alignment, because Jurisprudence is fundamentally a principle of Balance, not Law, in that system, and there's almost nothing to go on for how something like the 5e paladin oath should be dealt with.

In any case, if you are not the DM, ask your DM how the metaphysics of ethics and jurisprudence are related in their game. If they are related, seek to focus more on the adjudication of ethics than jurisprudence as the ethical component is probably dominant in most GM adjudications. If they are not related, ask about Jurisprudence in general, and the adjudication of your class abilities in particular, if necessary, focusing on what an Oath means, what it means to break it, and how reparations might be made if the oath is indeed broken.

If you are the DM, I strongly recommend you consciously decide what philosophical system of justice you will use in your campaign, as all of the other options amount to unconsciously using some system and putting some introspective thought into this choice before you start running the game is likely to make these kinds of decisions much easier and much faster during the game sessions, since you will spend much less time figuring out how to approach each specific case.

If you want a place to start looking at different approaches to the philosophy of oaths, I recommend this paper.