[RPG] create a ‘sleeper agent’ if I cast Modify Memory on a target under the effect of Geas to forget the Geas

You are overthinking this especially since it's backstory, not a real situation in play.

A 17th level wizard has a +10 Will save base. (He loses a level from the rez, but it's still +10.) That's a 25% chance to fail the DC 15 Will save, assuming he's not super wise (most wizards that embrace lichdom aren't). So... He failed it! Done and done.

You're depicting a past event, so why does it matter? How does it change things if his chance to fail is 25%, or 50%, or 95%? It's never zero or 100% (since 1 always fails and 20 always succeeds) so you are trying to push precision into something that doesn't need it.

If you're looking for background color, then sure he "cast various spells to lower his Wisdom/Will save/resistances/whatnot," though it's mostly impossible RAW to land them while he's dead (readied action or quickened before slapping on the helm, perhaps). Or slapped another cursed item on him to that effect (vary something like a robe of powerlessness to hit Wisdom instead of Int). But in the end, it really doesn't matter what the real number is - you are as the GM just saying "he failed," whether he had an effective +20 Will save or a -10 Will save.

It's easy to get stuck in a rabbit hole as a GM - you have a lot of more important things to prep, a lot of things that will make real different to your players' enjoyment of the game, than this kind of obsessive detail. You are effectively making your game worse by choosing to spend your time in the hole. Pull yourself out, write the backplot, and move on.

Your questions:

1. Can a character under the effects of Geas cast Remove Curse on herself? YES.

2. Can this be prevented by saying "Bring me 100gp from the offering plate every day and don't cast remove curse on yourself?" NO.

3. How does this work? SEE BELOW:

Question 1:

There is nothing in the spell description stating that the creature is magically compelled to do, or to not do, anything, even disobeying the command you have given. What there is is a consequence for disobeying, which it explicitly states is possible:

While the creature is charmed by you, it takes 5d10 psychic damage each time it acts in a manner directly counter to your instructions, but no more than once each day.

So YES they can cast Remove Curse on themselves. or in fact do anything else.

Question 2:

The geas spell description (PHB p.244) states that

You place a magical command on a creature that you can see within range, forcing it to carry out some service or refrain from some action or course of activity as you decide

This means that you can tell them to do something or tell them not to do something, but not both. Also it is singular, you can refer to only a single course of action. So you can't command them to not cast Remove Curse on themselves (or anything else) as well as the primary command.

So NO you can't prevent them from casting Remove curse on themselves in addition to the primary command.

Question 3:

I am going to rephrase the "How does this work?" question to something more specific: At what point does the target of a geas know it has had a geas cast on it?

The spell description states:

If the creature can understand you, it must succeed on a Wisdom saving throw or become charmed by you for the duration.

and that

A creature that can't understand you is unaffected by the spell.

This means that they have to hear and understand your verbal command to be able to follow it, there is no "telepathic" magical transference of meaning. This means the creature effected knows at least that someone has given it a command it can hear in a language it can understand at the time of casting. As stated earlier there is no magical compulsion to follow the command.

Do they know they have had a spell cast on them? The Rules answers 2016 state, using suggestion (PHB p.279) as an example, gives strong guidance:

You’re aware that a spell is affecting you if it has a perceptible effect or if its text says you’re aware of it (see PH, 204, under “Targets”). Most spells are obvious. For example, fireball burns you, cure wounds heals you, and command forces you to suddenly do something you didn’t intend. Certain spells are more subtle, yet you become aware of the spell at a time specified in the spell’s description. Charm person and detect thoughts are examples of such spells.

Some spells are so subtle that you might not know you were ever under their effects. A prime example of that sort of spell is suggestion. Assuming you failed to notice the spellcaster casting the spell, you might simply remember the caster saying, “The treasure you’re looking for isn’t here. Go look for it in the room at the top of the next tower.” You failed your saving throw, and off you went to the other tower, thinking it was your idea to go there.

Breaking this down you’re aware that a spell is affecting you if:

• it has a perceptible effect
• if its text says you’re aware of it at some point

You might miss that a spell is affecting you if:

• the spell is subtle
• you failed to notice the spellcaster casting the spell

Applying this to Geas:

• it does not have a perceptible effect at the point of casting and
• the description does not explicitly state you are aware of it.
• The damage done by the spell is psychic, in the mind of the target and that only happens at the point you go against the command and The spell has only a Verbal component, no handwaving or bits of fleece so it is subtle (a velvet wrapped hammer springs to mind)

So it is down to whether the character notices the spell being cast as to whether they know that a spell has been cast on them, up until they take damage and at that point it is whether they recognise the spell effect for what it is.

The PHB p203 describes a Verbal component of a spell as:

Most spells require the chanting of mystic words. The words themselves aren’t the source of the spell’s power; rather, the particular combination of sounds, with specific pitch and resonance, sets the threads of magic in motion.

This means that the words of the command given are not the only part of the Verbal component, at the very least the words have to be said at a particular pitch and resonance etc. So it will be down to the DM to set the difficulty for the target, or any other witness, to notice and recognise that a spell has been cast.

The target has to be aware of the command so no perception roll is required so as a DM I personally would base it on an Int(Arcana) roll (or Wis(Arcana) if you, like I do, use the alternate rules on skill characteristics with a different take on the information provided by a success). I'd suggest a difficulty of around 13 depending on the circumstances.