There are a number of different strategies one can take (besides the possibly obvious: remove material spell components/focuses/holy symbols/etc.):

1. Make the jail special
   

• Suppress sound (e.g., permanent or ongoing silence) to eliminate verbal spell components

• Suppress light so that the spell caster cannot, for example, target individuals or locations via line-of-sight (e.g., use illusions over doors or windows to block line of sight).

• Suppress divination (e.g., permanent or ongoing nondetection) so that the spell caster’s buddies cannot scry in order to pop in via, say, teleport, and rescue them.

• The jail is built in an area that "naturally" suppresses, hampers or distorts magic.

• The jail is constructed using permanent or ongoing anti-magic spells

NOTE: Following up on one of SevenSidedDie’s comments on the original question, spell caster-proof jails may be rare enough and expensive enough things, that one would not expect small towns, or even necessarily any particular city to have the capacity to hold a spell caster (although this will depend somewhat on how common spell casters are in the campaign setting, for example, if every small town has a few high level casters, then these kinds of jail enhancements might be common). This kind of scarcity might make the possession of a spell caster-proof jail a special resource for a region, or even between nations/principalities/city states/etc. where extradition treaties, rendition treaties, or even trade treaties are negotiated around arranging the incarceration of a spell caster (imagine a side narrative about arranging the transfer of a incarcerated spell caster: fun!).

2. Alter the incarcerated spell caster
   

• Magically compel the spell caster not to use spells while incarcerated in jail (e.g., a geas, some really nasty custom curse, mark of justice, etc.).

• Make the spell caster wear some difficult to remove (probably homebrew) magic item that suppresses somatic movements, sounds, or even magic.

• Tag the incarcerated caster with difficult to remove tracking and summoning magic, and make them aware that there will be sanctions for jailbreaks.
  

• Place the spell caster in some kind of suspended animation (e.g., the vanish sink spell of 1st edition AD&D).

• Drug the spell caster so that they are incapable of casting (per orlp’s excellent comment).

3. Have the jail include guardians who are practiced at containing spell casters.
   

• Abjurationist wardens

• Mage-slayer wardens (perhaps equipped for and specializing in non-lethal combat and restraint)

• Wardens equipped with anti-magic devices

• Wardens that are magic-suppressing, magic resistant, or magic immune monsters or constructs.

4. Make compelling appeals to the incarcerated spell caster’s reason, morality or emotions

• Threaten the spell caster’s familiar, family, colleagues, or benefactor with sanctions (e.g., your family pays a huge fine if you jail break; your familiar will be destroyed if you jail break; your teacher will be stripped of standing in the college of magic if vouchsafed spell casting students jail break; your first born child will receive the spell caster’s sentence if they jail break; etc.).

• Appeal to the sanctity of law (if that is something that the spell caster cares about).

• Escalate the legal threats against the spell caster by compounding an original crime with a jail break (e.g., the punishment originally was 6 weeks in the clink and a small fine, now it is the amputation of your right hand).

• Make plea deals, so that publicly the spell caster is seen as punished, but the powers that be arrange to trade benefit for leniency in actual sentencing.

Telekinesis, a level 5 spell, says you can manipulate an object. I would think that the example of opening a door is at least as complex as pushing a button on a rod.

You can exert fine control on objects with your telekinetic grip, such as manipulating a simple tool, opening a door or a container, stowing or retrieving an item from an open container, or pouring the contents from a vial.

Also it does not say that you cannot manipulate a magical object. So I think this one is valid.

Unseen Servant, a level 1 spell, should also be able to do it.

The servant can perform simple tasks that a human servant could do, such as fetching things, cleaning, mending, folding clothes, lighting fires, serving food, and pouring wine.

Again, there is nothing saying that it cannot activate a magical item. (However, it would not be able to read a scroll or drink a potion)

However, you say at a distance and the Unseen Servant needs to be able to reach that location. Some people say the Unseen Servant can hover, others say it cannot.

Ten Foot Poles could also be used to push on a button. You do not specify the distance, but there are no reasons why that would not work. Only it may take a few tries (i.e. Dexterity check on a fairly high DC such as DC 16.)

Missiles could also be thrown at it (rocks, arrows, bolts, etc.) However, that may be quite a feat to hit that tiny little button (although a giant can throw big rocks, he! he!)