One of my players is a demonologist and requested a cantrip related to the field. So I took a crack at it and came up with the following thinking that an attack on the mind was fitting:
Three-Word Curse: Nails
Enchantment cantrip
Casting Time: 1 action
Range: 60 feet
Components: V, M (a fragment of ram horn)
Duration: Concentration, up to 1 minute
You speak in a demonic tongue slowly cursing the mind of a creature you can see within range. If the target can hear you (though it need not understand you), it must succeed a Wisdom saving throw to end the spell. On each of your turns for the duration, you may use your action to have the target attempt the Wisdom saving throw again. The spell ends if you use your action to do anything else. On the third failed attempt by the target, it experiences the feeling of several nails piercing their skull simultaneously and takes 5d8 psychic damage, and the spell then ends.
The spell's damage increases by 1d8 when you reach 5th level (6d8), 11th level (7d8), and 17th level (8d8).
Does the risk of wasting the invested time balance the high damage reward?
Best Answer
The risk/cost is too high
The cost is:
This is a rather high investment to start with, but there is also the risk:
Below level 5 the damage potential of this spell (5d8) is higher than other cantrips (3d12 with toll the dead is the next), but on higher levels it seriously starts to lag behind. Other cantrips also get +1dX, but they can be used 3x as often.
Add to this the aforementioned risk. Requiring 3 saves already gives this a low chance to succeed, but to achieve that on 3 consecutive rolls plunges the chance of success even deeper. The effective time/action investment will be way over 3 because of needing to restart on any successful save. This spell will extremely rarely do anything, especially considering that most combats do not last for even 10 turns (usual is 5-8 in my experience).
All in all this spell is so risky I would dare to call it near useless and the damage progression is miscalculated.
Recommendations
If you want to preserve the "three word" aspect, imitate spells like contagion and do not require the failed saves to be consecutive, but end the spell with 3 successful ones (also not necessarily consecutive). A mechanic resembling death saves. And either do not require any further action investment (like other such spells) or make it a bonus action. Also, I would advise you to make it a higher level spell, not a cantrip.
If you want to just make a more thematically appropriate cantrip, reskinning an existing one would be much easier to do and to balance.