After reading through the description of the spell compelled duel, it seems as though the spell does not actually say what it does with respect to restricting the movement of the target creature.
Let's review the description bit by bit:
You attempt to compel a creature into a duel. One creature that you can see within range must make a Wisdom saving throw. On a failed save, the creature is drawn to you, compelled by your divine demand.
"Compel a creature into a duel" is the flavorful description of what this spell does. The mechanical nature of this is supposed to be defined when the spell describes what happens on a failure. What follows after "on a failed save" is the spell's definition of "compelled into a duel".
So what happens on a failed save? "The creature is drawn to you, compelled by your divine demand". This definitely reads like more flavor text. I can see two interpretations of this phrase. The first, we can understand "drawn to you" to mean "the creature cannot willingly move away from you". But we can be certain that this is not the case, as the next part of the spell tells us what happens if the creature does attempt to move a certain distance away from you. The only other reasonable interpretation I can see is that this phrase is, once again, more flavor text, and the spell is going to tell us what it does later on in the description.
The next bit is perfectly clear,
For the duration, it has disadvantage on attack rolls against creatures other than you,
Moving on.
This where it gets weird. The spell now gives a condition which triggers a saving throw, and defines what happens on the success of that saving throw:
[The target] must make a Wisdom saving throw each time it attempts to move to a space that is more than 30 feet away from you.
Okay, this seems to heavily imply that the target is able to attempt to move to a square more than 30 feet away from you. This is what invalidates the first interpretation of "drawn to you" mentioned previously.
Now the spell describes what happens on a success on the saving throw:
if it succeeds on this saving throw, this spell doesn’t restrict the target’s movement for that turn.
As written, it seems the spell already does not restrict the target's movement because the spell never says anything to that effect. It does not describe what happens on a failed saving throw. It never tells us how it restricts the movement of the target.
But it gets worse. Because the spell never tells us what happens on a failed save, there is no reason given in the description that would stop the target creature from attempting the saving throw until there is a success.
What does compelled duel actually do?
This question seems related, but is muddied by the fact that the asker employed an outdated or incorrect printing of the spell description.
Best Answer
The spell is unclear, so it will be up to DM ruling/interpretation.
As you pointed out, the spell is unclear. I haven't found any kind of clarification on it in the Sage Advice Compendium and the spell has never received any kind of errata.
And, well, spells do what they say, so, failing the saving throw does... nothing. From my reading. It does not stop the target from moving outside the range.
How would I rule it (a.k.a.: What I think is the intention of the spell)
Obviously, the RAW interpretation seems far from the intended - otherwise, it would just not include any of this saving throw on movement at all. The intention when I read it is, basically, assuming the consequent. On a successful save, the target can move. On a failed save, the target can not move. This intention, to me, is clear from the text you call flavor1. Although I agree that it is not mechanical, it does state the intention.
You also mentioned (in chat) the possibility that the target starts outside the 30 ft. range anyway - which is possible through actions that move him (e.g. being shoved away). So, the way I would rule it is wording the spell somewhat as follows:
That way, if the target is already at, say, 35 ft. from you, it can freely move towards you and, I would rule, even in a circle maintaining its distance. However, it could not move away from you without the saving throw.
Ruling that it can only try once is also logical - as movement is only spent after, well, the target actually moved. If it could attempt more times, then it would eventually succeed, which makes the spell again pointless and doesn't make sense to me.
1 As noted in the comments by MivaScott (thanks for the link, I knew we had that question somewhere), spells in 5e do not contain "pure flavor". As I said, in this case, although the text does not describe the specifics of the mechanics, it can not be ignored either, hence my point that the intention of the spell is clear, we just need to rule how the mechanics actually work.