Combine the Necrotic Cyst line of spells with Hindsight. Investing in items of necrotic cyst, necrotic scrying, and one or two hideously expensive polished steel and diamond blocks (nothing breakable, mind you) of hindsight would make any rather evil but almost annoyingly hard to corrupt police force proud.

Start up your reign of spying with listening coins and spymasters coins for basic surveillance techniques, casual police stationed in the streets for detect thoughts, enchanted beds of dream sight for traceless spies wandering around, and the gem tracer spells for monitoring established areas.

---

A quick summary of the spells and items referenced in this answer:

• Necrotic Cyst (line of spells)
  • Population control and traceless scrying. Requires a policy of implanting population with Necrotic Cysts.
  • Available from Level 2, good to establish domination over a town.
• Hindsight
  • The ultimate forensics spell. Squashes conspiracies.
  • Available from Level 13+. Absurdly expensive. Worth considering a spell-to-power erudite for this one spell in lieu of sublime chord.
• Spymaster's Coin and Listening Coin
  • Available at low levels. Great cheap scrying sensors for bugging people you know are plotting against you.
  • Doesn't scale to population
• Distilled Joy/Distilled Pain
  • Available with some money. BoED/BoVD. A way to fund your city, properly. Only include if you need a source for all of these items.
  • Kind of obvious.
• Susurrus of the City
  • Urban Druid (very obscure spell list)/5
  • Answers questions about anyone located in the city, to the best of the city's knowledge.
• Identify Transgressor
  • Answers any question with a name for an answer. Have a "jury" of 7 clerics who spend their days casting this.
• Gem Tracer
  • Gain reusable scrying foci. Scry on target object as standard action (no cost), locate object (for ever, and for free)
  • Great for identity cards.
• Mind Probe
  • Telepathy/5
  • Does what it says on the tin. Once you capture the rebels, you can figure out why they don't like your benevolent rule. Then implant them with a cyst and dominate them into being fixed.
• Modify Memory, Psionic
  • Telepathy/4
  • We totally didn't do the prior mind probe. These aren't the secret police you're looking for.
• Hypercognition
  • sometimes the threats come from outside. This is the way to identify quest targets.
• Amanuensis
  • Necessary for transcribing reports into books
• Scholars touch
  • The ability to read books in a round, allowing for efficient information processing.
• Call to mind
  • The ability to retrieve read information, creating a colourable excuse for the prior stuff to work for management of a city and secret police.

Magical items: any of the above enchanted into an item.

Other answers in the theme of magical cities and evil:

• Lowering will saves through mind control.
• Living forever in D&D
• Investigations in D&D
• Dropping clues of corruption

Restate our assumptions

This sort of game isn't a historically accurate fantasy game. With a city that will be this powerful, the game itself should be relatively high optimisation. The city needs to have reasons for a secret police, otherwise there's no need to spend this kind of effort on the task. Beyond that, the capabilities in this answer presume a relatively high level of information processing capability. (See the book IBM and the Holocaust for details on why information processing is a necessary precondition for massive institutionalised evil.)

However, presuming that this kind of ahistorical situation exists, and the city exists in an ideologically charged world with a nominally "backsliding" populace, these kinds of secret police are quite effectve at enforcing the pretense of ideological change. Shaping the thematic elements of the setting to be equivalent to a post-war, ideologically riven Europe, is also an excellent fantasy basis. Take a look at Max Gladstone's books for this kind of true urban fantasy workup.

I'm also presuming that alignments don't really matter, since "detect evil" rather spoils the conceit of the game. The number of people ... doesn't really matter, though there are some efficiencies of scale. Looking at the historical picture suggests a ratio of 1:400 is normal for soviet times, not counting informants. Looking at penal colonies, like Caldoche, we can infer a guard ratio of about 500 to 12000 or about 1:24, presuming external imperial support and a very facticious population.

For these purposes, we'll presume an administrative ratio closer to 1:40, allowing for technological inefficiencies. This ratio would likely comprise the full military of the town, ultimately scaling to a 25,000 metropolis.

The character of the Mayor, his Staff, and his police.

We will presume leadership of true believers. These guys obey the maxim "the ends justify the means," and actually care about achieving their desired ideological conversion goals. The need for ideological purification drives the establishment of the secret police, and concerns about the old order and reactionaries, imperialists, hegemonists, etc... explain their presence.

The Mayor

To pull this off, we best need three characters. The party face, "Mr. Mayor":

Our mayor must be charismatic, charming, manipulative, and probably should be clean shaven with stressed wrinkles. Nothing says evil vizier less than slightly shabby clothes (in suitably heroic colours. No villain black here), no goatee, and an expression of overworked ... non-malice.

Our ultimate spell for police work is Bard 6, Sor/Wis 9 "Hindsight". It's got a 1k material component. This gives us two options (not counting archivist + cheesy divine bard). Bard + sublime chord or spell to power erudite (in the most conservative assumption of powers known.)

Given his nominal mayoral duties, Bard + Anima Mage + Mindbender(1) + Sublime Chord is quite appropriate. Bard is obvious in this context, and allows him to start his plot by Level 3 with the Necrotic Cyst feat. Anima Mage is important because of the cheap spying it provides. His two binds are going to be Naberius for public speaking (taking 10 on diplomacy as standard action can be quite useful for calming people down) and spying via command + disguise self and Malphas for Birds Eye viewing, allowing a scout at level 4 that can sit on windowsills and peer into rooms. Sublime Chord is for better spellcasting, due to an early need for hindsight.

Effectively, this auditing apparatus can be completely effective by level 13. Spot checks with the bird, Hindsight to investigate any known subversive meeting places without fail, and Necrotic Cysts implanted in people to use them as hidden spies and dominatable (without need for line of sight) patsies. Beyond that, regular dinner parties with a random selection of the city's population (a good census is critical) should allow for effective sense motive checks combined with a detect thoughts in the background to get a sense for where the hindsight spell needs to be targeted.

With the correct city building codes, no wall will be able to be able to block the mindsight via the mindbender's telepathy. Random wanderings of the city should provide for suspicious gatherings for subsequent investigation.

The critical feats for Mr. Mayor are: Necrotic Cyst and Mindsight.

His job is to establish the network of cysted individuals and ensure that the majority of the non-ideologically backwards population supports his goals. Frequent, significant, feasts are his style. Attendance is drawn by lot from all people in the city (completely random, I assure you, and is absolutely voluntary. He'll even send people around to tend you if you're ill and can't make it.) Here, glibness, detect thoughts, improvisation, and the insidious insight spell all work to supplement an evening of glad handing and sense motive. Most people will be put at ease: the food is good, no arrests are carried out, and everyone is made to feel loved by the state. Enjoyment will be had. By order.

During the day, he will maintain and implant the network and records of the citizens with necrotic cysts (all criminals and people of power, eventually growing to a mark of citizenship). As his levels grow, so will his control.

The town ideologue

The archivist can serve as the town ideologue. Rejecting the false theurgy of the old bourgeois, the archivist can usher in a new era of right thinking and prosperity. By order.

Her job will be to maintain and craft the divine half of the magical items to be used for the town's benefit, figure out problems of ideology and other research. Her job will also be to maintain the records of the citizens, keeping all the reports current and making judgements as necessary. We will gloss over the need for finding the divine spells (urban druid especially), by passing the buck and saying that the empire that founded this city in contested lands is handling that problem. If there isn't a special library for the archivist, we can replace her with a team of an urban druid and a city domain cleric, but that is less... cute. She will, of course, have the Necrotic Cyst feat, because there's no reason why she shouldn't be able to dominate any of the citizenry at no notice.

Special attention must be paid to the spells Susurrus of the City, Identify Transgressor, and Expose the Dead. At level 7, with access to identify transgressor, most crimes found in the city can be punished. Expose the dead at least provides level 3 crime scene forensics for an ad-hoc solution to this problem. Susurrus of the City, at level 9, allows the archivist to settle into her niche of finding out intelligence about the city -- by asking the city. She will have developed a checklist of daily questions allowing for the efficient identification of threats to the city. Enchanting this into a device is top priority due to the frequency of communion. She will also be responsible for establishing the taxation office (discussed later), corrections office, and food centre.

The divine variant of anima mage is also an interesting choice here. I personally like it as a thematic link between the characters, and it provides significant low-level options to the budding ideologue. At the cost of a single spellcasting level, it technically isn't worth it, but I'd recommend it.

Judge Fred

Judge Fred is a spell-to-power erudite. I would accept the number of unique manifestations per day limitation here in exchange for the note of levelling up in other psionic classes being limited to base classes only. I would also multiclass him binder, again, to maintain the theme, but to keep his options open throughout the days of judgement.

Her job is to be the inquisitor, the leader of the city's police forces. Being able to take mind probe and object reading is critical for proper forensics, and she is happy to take advantage of the knowledge of the completely reformed reactionaries currently pondering the errors of their ways back in the new glorious republic. Between seer powers, telepath powers, and strategic spell-to-powers (taking advantage of spending power points instead of absurdly expensive material components), Judge Fred is nominally weak, but extremely good at finding out information. Again, Level 13 is the sweet spot for Hindsight, allowing a comprehensive investigation into the city to begin. She would also take Necrotic Cyst (adapted for spell to power), just to fit in.

She will also manage the library of Gem tracers, enchant spymaster's coins at low levels, and deal with hypercognition at the highest levels. She will also spend her spare time enchanting as appropriate.

Feats wise, she should make sure to take earth sense and earth power, both of which are useful, and in general invest in power point cost reduction feats.

They will live in a tasteful, subdued... monument to the people's glory.

She won't tell anyone why her name is Fred, or what it's short for. Given that asking tends to lead to disappearances or other corrective activity, no one has ever asked twice.

On Cysts

Necrotic Cysts can be implanted in "living creatures". Therefore you can achieve your optimal panopticon via caged mice or glassed oozes or equivalent. Enchanting a mirror with necrotic scrying, which is a hilariously cheap spell (Sor/wis 2) will allow using it to use the subject's eyes xor ears. It's also worth implanting prominent citizens with these cysts.

At higher levels, necrotic domination makes it trivial to dominate the important members of town "to protect the children," if necessary. Most days, however, necrotic domination will be spent dominating telepathic creatures with mindsight. These creatures can detect thinking beings in their radius, and more importantly, can be told to query beings they suspect of suspicious activity. A bad answer or not obviously following instructions will lead to immediate and completely voluntary happiness intensification.

The security force, or Stasi++

A police function is almost always an audit function. It's not so much catching people in the act (though it's nice when it happens), but the forensic work necessary to prevent someone from breaking the law again.

At the end of the day, most of the citizenry will be fanatically dedicated to Mr. Mayor, there will be virtually no crime not allowed by him, and he will have knowledge of the city's workings.

The police force simply needs to have portions of the builds above. Level 1 binders get birds to spy on people, which scales wonderfully, Bards get fascinate and glibness and mindbender -> mindsight. And since every policeman carries a cyst, it's trivial to ensure their loyalty (especially if you are in charge of the two or three "rival" organisations trying to take you down.).

The only real problem is defending the city from external threats. But, given any organisation in the threat, Mr. Mayor can simply "convince" the leaders to be more receptive to his needs. Unorganised threats, unfortunately, require adventurers.

The Stasi are, in this instance, the force to be emulated:

One of its main tasks was spying on the population, mainly through a vast network of citizens turned informants, and fighting any opposition by overt and covert measures including hidden psychological destruction of dissidents (Zersetzung, literally meaning decomposition). It also worked as an intelligence agency abroad, the respective division Hauptverwaltung Aufklärung was responsible for both espionage and for conducting covert operations in foreign countries.

To be clear, these spies were not very nice people. Ideologically driven, granted secret power and a thankless job, most of the time spying on people who weren't very objectionable. For a host of NPCs, watch The Lives of Others. (Or, for an even more sinister feel, watch The Prisoner (1967).) The objective of these forces, narratively speaking, is to force the players to wonder "are they doing more harm than good?".

With magic, however, the job of the secret police is much much easier. The primary job is to locate, identify, and sanitise threats. At the epitome of the arc, the Hindsight spell (appropriately widened) can provide detailed and traceless transcription of all conversations and actions within a 120' radius for over a week. This makes plotting of any kind extremely difficult, since silent gatherings of people using telepathy can be instantly identified and all other conversations analysed for potential problems.

Keeping the populace in line will actually be fairly easy, since a decent PR effort is just common sense. (Especially as, after some time, most of the population will have figured out how to pay taxes via strategic bliss and will likely be addicted to it.)

Keeping adventurers in line will require second order agents provocateur. It's important to have signs of an resistance movement (along with a thieves guild) in town, as lightning rods for discontent. The first order of business for these lightning rods is to use the energy of the people attracted to them against useful targets, either actual dissidents in the city, or against external threats for whatever reasons can be fabricated. This is where the other beauty of hindsight comes in, as it's not an active divination, most protections and detections against divinations are moot. Outside of that, all public guards should have arcane sight enchanted into them, and anyone carrying objects marking them as very high level should be treated with the utmost respect.

Digressions aside, the growth of the town's Stasi should reflect the population needs with a strong ideological "devotion to the state" component.

The difficulties of information processing

The inherent challenge here is that there are no ways of tabulating data quickly. On the other hand, with the spell Amanuensis, reports can be collated quickly into books and read instantly by Scholar's touch. If enchanted into an object, call to mind should allow for the reasonable accession of this knowledge by either the ideologue or judge. Note well that the books should be significantly sized as scholar's touch operates per book, rather than per page. With a high autohypnosis, a precis of this reading can be cemented into mind, for future work.

Beyond this collation mechanism, (and again, assuming modern technological ideas), the principles of relational algebra are of huge importance here. So long as creatures with a minimal (4-5, basic language recognition and counting, no comprehension needed) intelligence can be bound, it's possible to map relational algebra operations across a large series of creatures. So long as they have no free will, the benefits of punch card or database operations can be realized with the appropriate undead, elemental, or outsider army. Just significantly slower. Still, the capability of being able to perform these operations makes the modern police state possible.

If an efficient source of these creatures can be found, they can also serve as transcribers of information, taking surveillance intake and turning them into written logs of activities.

The benefit and challenges of Hindsight

Hindsight can show events in the past, at different temporal resolutions. Each resolution, day or week (longer resolutions aren't relevant to this task), is unit per caster level. Usually the judge should be managing casting of this spell, as she can subvert the material costs. It's also relevant because linked power can subvert the hour casting time.

Borrow here the US's process of parallel construction. Any information uncovered through traceless scrying techniques can plausibly be discovered through discoverable information gathering techniques. As the populace (and adventurers) don't know the source of the knowledge (and anything that cannot be reconstructed cannot be acted on publically), the secret should be quite safe for quite some time (barring divine intervention or equivalent information gathering capabilities by the opposition)

The city underground should be studded with a grid of hindsight chambers such that all spaces are covered by at least one of them. These should be as secure as possible, and boring save for the sheer amount of them. Unfortunately, there will need to be around 2000 of them to cover the entire city, and the resulting item is easily epic. Therefore, as the normal unwidened spell will require twice as many listening rooms as there are people, a different approach is necessary.

The first approach must necessarily be subsidised housing. By going to roman style tenements, the city can grow up instead of out. It's also much easier putting listening chambers in the middle of what amounts to an apartment block than it is underground. Have the city focus on towers surrounded by green spaces instead of the normal flat construction. Since wall of stone is simply a good investment for a city to have in a wondrous item, (and since that will allow for good spots for listening devices to be emplaced as well, see: construction of us embassy in moscow). These buildings can be rent reduced to enhance desirability as well.

Adventurers at the gates: Access Control

By Level 13, implantation of a necrotic cyst (internally, of course, to reduce the odds of detection), is a hidden prerequisite of town citizenship. Guards at the gates will have items of detect cyst (as well as the more mundane arcane sight and detect thoughts), and use it to direct non-citizens into an enhanced questioning line. Adventurers will be assigned guides and informed of the rules of the city. There is a one time head tax of 100 gp to purchase a 1 year visa into the city after a (private) search of belongings. It will be made clear that the visa extends to the adventurer and his/her current gear, and that significant changes in equipment may invalidate the visa.

There will be a scrupulously honest inspection of their goods, contrived to go through an occluded conveyor for at least a round. (Most of the inspection process will itself be open to inspection by notoriously touchy adventurers.) For each character purchasing entrance, a gem tracer spell will be linked to their most obviously expensive magical item.

If a person presents previous identification papers (signed by arcane mark), the gem linked to those papers will be found, and the object in question located. If there's an asset mismatch, the visa will be deemed invalid and a new visa (and extra scrutiny) can be purchased. This way, the city has a rough idea as tho the most capable item each adventurer holds and where it is, allowing trivial scrying to be performed to thwart any unauthorized planning. If the adventurer leaves the item behind, it is, at the very least, a handicap.

Adventurers lacking magical items will still be provided the visa, and the gem will be attuned to the visa itself. While not as reliable, this at least provides two-factor document authentication against the user.

Adventurers bringing fake visas in will be relegated to the taxation office until their corpses are ready for reanimation.

Bringing a town together, The portrait of a city

From the perspective of Mr. Mayor, the town will face four vectors of stress. Internal competition, adventurers, resources and external threats. Happily, they can be used against each other.

A note on resources

The things I suggested above are expensive. As a consequence, this city's laws will be extremely lawful evil, simply as a profit making source. Anything that damages the city's capabilities is against the law. Restitution is to provide a higher upside than downside. Thus, murder is bad because it deprives the city of labour. Restitution can be provided by resources in kind to the city on top of resources in kind to the dependents (which would otherwise be a drain on the city.) Theft is bad, see above. Breaking contracts is bad because it encourages people not to live in the city. Etc, etc. Every crime, very specifically, has a monetary value attached to it, to represent the damage it causes to the city. Corruption is one of the nastier crimes, as control fraud can hollow out the city and make it not be as effective. Emphasis on the Lawful aspect of Lawful evil.

Thus, the city needs to be a nice place to live. In one of the city squares, therefore, there are three major institutions: the Joyful Tax office, the corrections office, and the food bank.

The food bank will have (at least one) item which casts Sustain (BoED) on people. Sustain takes care of their hunger and water needs for at base caster level. It targets (at CL 8) 4 people per cast and lasts for 48 hours. It's necessary infrastructure for the city to survive sieges, and can eventually allow farmers to be in more productive positions. It's worthwhile getting an extended version of this put in at CL 10, which causes the spell to effect 5 people for 5 days, instead of 4 people for 48 hours. The charge per use should be 1 GP, as that undercuts the cost of food by 1 sp. (3 sp per day for common meals by 5 days is 1.5 gp. This means that people can spend the extra silver how they wish, likely on fancy banquets for fun. See the philosophy behind soylent meal replacements). With a population of 4000, all of whom are encouraged to make use of this resource (mainly because of the discount), this item will have to service 50 people an hour for 16 hours a day, which is quite reasonable. It will produce an income of 4000 gp per 5 days, and will therefore pay for itself in half a year. More deluxe models can be installed with the profits.

The Joyful taxation office is the cornerstone of the city. The city must require that its taxes are paid in XP, rather than gold. Potable XP is generally found in either Ambrosia (BoED) or Liquid Pain (BoVD), and the city isn't fussed about which it accepts. It issues a currency to represent the potable XP, and people can make money at the joyful taxation office. It contains rooms set up to cast distilled joy and distilled pain on people, as well as an item of restoration to patch people up after pain sessions. Spellcasters are available to hire to generate blissful states. A series of "complimentary" iron maidens of preservation (BoVD) and pain extractors are available on the other side in silenced rooms.

The cornerstone of the taxation office are criminals who are unable to pay off their crimes. They are placed in the pain extractors (as few qualify for distilled joy, though no bias is enforced save for the capability of producing the appropriate resource) until they generate sufficient liquid pain to pay off their debt to society.

The corrections office doesn't maintain cells. Incarceration is wasteful. Instead, it implants necrotic cysts into people and uses necrotic domination to walk them across the street to the tax office, as well as managing minds of the criminals assigned to the maintenance jobs that lucky (or connected) criminals can aspire to.

I would, at this point, encourage a house rule to allow a neutral substance "Distilled concentration" to go along with the good and evil substances, in much the same lines. Again, the emphasis of the city is on self-preservation, rather than evil for evil's sake. Only for those poor souls unable to muster sufficient concentration or joy are forced to go into the maidens. There should be acceptable spells at the conclusion of service to fix most of the mental and physical problems as well, as that is simply preservation of the workforce. For the people who die in the city, they are, of course reanimated (unless they pay the city more than the worth of their corpse) to do the really icky jobs that no one wants to do.

External threats and adventurers.

Dealing with adventurers is tricky, but at the end of the day, so long as you have things to point them at and shiny things to buy them off with, most won't really question the "evil" nature of the town, especially as some areas will be not mentioned in their presence. Given the resource production of the town, most adventurers can be bought off to go attack external threats.

Yes and no

This is what the ability gains every four levels represent: actions your character is taking to improve an ability score. Most players don’t spend a lot of time discussing why they get those gains (since it’s “obvious” why the barbarian gains more Strength or the cleric gains more Wisdom), but the function of this mechanic is precisely to model the kinds of training that you are discussing.

In general, the game leaves the description of downtime training, maintenance, and so on very vaguely defined; that is supposed to be up to the players to describe. But leveling up isn’t magic. It’s an abstraction for exactly that kind of training and practice. You are assumed to be constantly studying, or lifting, or practicing skills, or maintaining gear, and so on, during any downtime you have. It’s all abstracted away, and then at some point it “all comes together” for you, you have that eureka moment, and you level up.

(No, this system does not bear close scrutiny.)

This is an extremely slow way to gain any ability score, however, and the game’s math has it “already accounted for,” so using it to increase Strength as a wizard is, from the game’s perspective, basically the same as losing Intelligence – since the game assumes that’s where you’ll be putting it (and monster designers take the resulting save DC increase into consideration when deciding monsters’ saving throw bonuses). But you still can.

Ultimately, wizards have exceedingly little use for Strength. A handy haversack and portable hole are great items anyway, so carrying gear isn’t all that important, touch attacks are trivial in most cases anyway (and you usually prefer ranged ones, which use Dexterity). The only real problem is the risk of being rendered unable to move by Strength drain or damage.