Political games require both excellent record keeping and a willingness to let players have excessive impact by comparison to normal gaming styles.
Taking my own prior experiences, the best working political campaigns were in L5R and Pendragon... both games where I've made extensive use of record sheets in various levels of detail.
The details needed don't include the combat capabilities of the NPC's most of the time. They do require tracking what secrets they have, what their resistance is to PC manipulations.
It also helps to make a map of the major players.
Further, it's important to distinguish a Court game from a Rulers game... they are very different in how you scope things out.
For a court game, the secrets are very important. Everyone collects dirt on everyone else. So everyone should have some dirt to be collected. And then it helps to have a grip on who likes, hates, or is sleeping with whom. And, as history shows, often, gender was no barrier to that last one, nor was hatred.
For a Rulership game, the kind of game where one or more PC's are rulers in their own right, the secrets are less important, but the resources in terms of troops, allies, and enemies are more so.
In either case, knowing who wants what and has what else to offer for it is important.
Political games are where a good information storage mode is vital... you can very quickly wind up with more NPC's than you can remember. But you don't have to remember; you can instead just make certain you can find it when you need it.
Likewise, it's vital to be consistent in who has what opinion about any particular other important NPC. Which is where the cross-reference table is wonderful:
\begin{array}{l c c c c}
& & \rlap{\text{About}} \\
\text{Who} & \text{Edmund} & \text{Franz} & \text{Corwin} & \text{Ld. Dunny}\\ \hline
\text{Edmund} & - & +5 & +3 & -5\\
\text{Franz} & +1 & - & -1 & +1\\
\text{Corwin} & +5 & +6 & - & +6\\
\text{Lord Dunny} & -1 & +1 & -5 & -\\
\end{array}
You can see that Dunny really dislikes corwin, but Corwin likes everyone. (That's why Dunny is unhappy with him... he thinks him an annoying suckup.)
Such a table is particularly useful for court games.
Likewise, if you number or letter their secrets, you can record their dirty secrets on a second grid showing who knows which ones for whom.
Also, keep in mind that allies need not be friendly... Just needing or wanting the same thing(s).
Further, depending upon the game system and style of play and GMing, the amount of prep needed can vary widely. In a strongly narrativist game, such as Blood & Honor, you start with blank map and relationship grids, and fill in as established by the group in play. In a GURPS game, a strongly simulationist system, it's far more likely to have the map populated, and the grid at start filled in, and let players modify from there. Either way, being able to find what you need to know is the most useful skill.
Each character should have a page of notes, as well... starting off, pretty slim. As the game progresses, note down favors done for or by them, and manipulations, alliances, and backstabs.
Don't forget that, in the absence of PC interaction, all relationships are subject to change as well. Provide a means for doing so "off camera." This could be as simple as GM Fiat ("I think it's time for Dunny to put Corwin out to the front lines...") or as convoluted as making rolls on their loyalty scores to each other, to see if there's a chance of raising or lowering the other's score...
But the single-most important element of a political game is that nothing survives contact with PC's unchanged. Any political action by a PC should have an effect. Not always a good one, and not always a direct one. Other parties who know what they did may react to it. THe person manipulated should have a reaction whether they succeed or fail... even if that reaction isn't evident to the players immediately. Failed blackmail becomes a secret. Failed or successful sexual liaisons become secrets. Revealed secrets become scandals.
So, why do it?
Because in a political game, a lot of it is pure roleplay. A lot more is making the efforts to get where you need to be. It's not all about the convincing. A lot of it is etting up plots and strokes that happen outside the scope.
In fact, in my own political games, the politics are more scenario-generation than scenario themselves. The nastiness of courtly politics leads to flared tempers, rage, and pinning murders on others in order to get them out of your own way. Rulership games involve marshaling your troops, fighting your wars when they happen, and dealing with threats outside of court.
If you can find copies, Birthright for AD&D 2E and the stand-alone Reign or Houses of the Blooded all have great advice on the political game.
You have the right answer in your choices.
Being a DM isn't about writing a script and continually nullifying player choices to keep them "on script". If you want to write a story without much outside input, then write fiction. Nothing wrong with that.
A DM is only one participant of the story when role-playing. Sure, typically the DM will set up the initial scenario and make certain decisions that will constrain choice. But you have to accept that once you turn the PC's loose on "your" world, they are likely to take your carefully crafted story, and fold/spindle/mutilate it (or all three!).
You can of course plan an overarching story for the campaign. It can a great idea to do so, it provides direction and focuses the campaign. But if the players make the story no longer possible and you can't come up with a satisfying way to "fix" it, then it's time to adjust.
If the party goes off of the rails, it is likely due to one of three things
- They notice the rails and want off! Whether by boredom or active malice they've decided to go left instead of right. Either way, you're getting feedback that your story is not as entertaining to them as you might have thought. Time to make changes!
- Player discovery - The players have discovered something cool about your campaign or their characters that you might not have thought about and want to explore it! This is awesome, it will mean that your story should go on the back burner for a bit. When the players show an active interest in the setting, nurture that, don't shut them down. Let it play out, it usually won't be a long detour and the players will rejoin the main plot-line with renewed vigor.
- Player Agency - One or more players has their own story that they want to explore. Quite often a player will build a backstory for their character that has some hooks in it. If you don't grab the hooks and work with them (I think we're all guilty of this at some point!) the player may still want to explore their half-dragon ancestry, or why they got kicked out of Star Fleet Academy. Let them run with it, rein them back in if it becomes excessive. In the same way that your story cannot dominate the table you can't allow one player to continually dictate what happens next. However there should always be room for each player to shine for a session now and again.
The last two are sides of the same coin. Both involve letting players "run the show" to a certain extent. One problem that I've seen multiple times is for the GM to plan one scenario after another after another, never giving the players a chance to catch their breath. Instead of plotting things so extensively, ease up occasionally. Let the players know that after the current big boss is killed that there are no immediate plans for the campaign and ask, "What will you want to do?" This can serve as a release valve and free up any building tensions.
Best Answer
I've centred on using an "Onion topology" for my campaign preparation. I also use a technique, both for campaign and adventure site preparation, whose name (not concept) I've borrowed from the crowd at Dream Pod 9 called "The Y-Cubed Law".
Onion Topology
One mistake I used to make all the time was lavishing waaaaaaaaaaaaaaaaay too much detail on things the players were never likely to find out and, thereby, rob my players of attention to details they were going to be facing constantly in the course of ordinary play from day one. (I blame I.C.E.'s old Campaign Law supplement for this since its step-by-step instructions started with the geography of a whole world including plate tectonics.) The cure, I later found, through much bitter experience, was to stop doing top-down design of my campaign and instead work on it from the bottom up. I later termed this "onion topology".
In onion topology you imagine your campaign world's information is arranged in layers like an onion. First you sketch out a whole onion in very rough detail – this would be things like gods, major continents/nations/whatever-suits-the-scope and other extremely broad-brushed details like that. You then move down to the core of the onion which is where the players are. That core is the immediate environs of where the players will start. In a typical pseudo-European feudal setting (which I will continue using as the example) it could be a single fief with attendant keep, village, outlying manor houses, etc. This you give in detail. You make plans for the keep and the manor houses. You make a map of the village. You place it somewhere in your "whole onion view" and draw up the local terrain according to the rough geography you have already put in place. You populate it with key NPCs (right down to the stats if this is important to your game). You find opportunities for plots via these NPCs. (Reading Georg Polti's The Thirty-Six Dramatic Situations is very helpful here!) You situate any typical adventure sites you want to use in the immediate future: haunted keeps, caves of despair, etc. Then, when everything is ready for your players ...
... you continue planning. Because players never do what you expect them to do. So you fill out the next layer of the onion to nearly the same levels of detail. Maybe you drop some of the finer details like minor NPCs, but you really need to detail one layer of information removed like the overlord's own fief, surrounding subinfeudated neighbours, etc. – anything, in brief, that the players might go to visit on a tangent.
Now you're ready to play. You have enough of a broad-picture view that your campaign isn't completely ad-hoc. You have many details for a smallish area in which you can situate your immediate-term needs for adventures and you have details for a larger area into which your players may suddenly sidetrack so you're not caught with your pants down. From here on in your campaign planning work is easy. Run your adventures. While you're doing that add little details to the next layer out until either your players branch out into it themselves or you're ready to branch out. As soon as they do, you start work on the next layer of the onion, always keeping your campaign details planning one layer removed from where the players currently wander around. This way you have the benefits of planning a campaign (especially the consistency) without the burn-out and the wasted effort. When your campaign finishes your detailed campaign notes will be a patchwork quilt of details and sketchy stuff, but your players will never know. To their eyes you'll have had this wonderfully huge and gloriously detailed campaign that they never reached the boundaries of.
The Y-Cubed Law
Hand in hand with the onion topology I used, since the early '80s, a technique I later grew to call "The Y-Cubed Law" courtesy of the boys at Dream Pod 9. Where the onion topology gives players the illusion of campaign breadth, the Y-cubed law gives the players the illusion of campaign depth.
What I found tending to happen both in my campaigns and others' was that in a bid to bring a place to life people would put in all sorts of interesting local quirks. For example: "In the village of Hothentot the temple bells ring once every ten days at 37 minutes past noon." Or: "In the land-locked city of Kamesh, right in the central square, there is an enormous, 1:1 scale statue of a whale." These are harmless little details of no meaning or merit; just flavour text, really. But ... players will always arrow in on these details and start asking questions. "Why does a land-locked country have a statue of a whale? How would they even know what a whale looks like?" ... and so on ad nauseum. As a player I always found the GMs floundering and looking, frankly, a little stupid as they get blind-sided by such questions and as a GM I floundered and felt, frankly, a little stupid when I got blind-sided.
Enter the Y-cubed law.
In the Y-cubed law, any time you introduce a detail into your setting (or, at its most extreme, into NPCs) you ask yourself the question "Why?" three successive times like this:
There. Now a piece of local colour has something that will stand up to a bit of casual curiosity. It also happens to provide some ideas for possible future adventures. (Where are the adventurers now that the villagers have the wherewithal to actually properly pay them? Were there any surviving bandits who'll come back for revenge?) The total time to ask and answer the Y-cubed can be, after some practice, thirty seconds or so for each detail and yet that minimal time and effort not only gives the illusion of depth but also gives the GM plot seeds that he can use in the future, thus saving time overall.