Plot Issues
1. Play out the sidequests anyway in the back burner
In a way this feels like a Shadowrun situation, and quite frankly if the players are chasing their own (maybe literal) ghosts, the factions are going to hire someone available to do it. Let the party handle their own stuff and a rival party gets the job. You can resolve the quest any way you feel appropriate for where the plot will develop, but now the players need to fight for their reputations. The next player plot hook could be buried in the rival groups.
2. Why not mix it up?
If the party is supposed to be a group of characters with high impact on their world, there is a way to make the character plots intercede with the core plots, even if they happen just a little off kilter from what you need to accomplish like the job is straightforward but Bob's old nemesis just can't see it go well, or George's scorned love demands a favor in exchange for forgiveness, or Nell's long lost sister might be in the mark faction's castle/town/whatever. While they may seem very divergent, the plot device has a setting for looping things together in the strangest of ways such that they all become related and keep the party together.
3. PC the NPC / NPC the PC
Do you have a specific way you want the quests to go? Follow one of the players on their sidequest (and anyone else with reason to be involved) and pre-gen NPCs for the rest of the characters to play - including a core enough personality to keep them involved. Meanwhile, the party does their quest. It might not be the smoothest accomplishment but it would then be up to you as the DM for embellishments but do reward the core party with the quest items for the players performing well in the side arc. This also gives you play room for fuzzy memory parts that you need to ad hoc or retcon.
4. Save this campaign for later
If the players are giving you enough plot to run a full game, it may mean that you should save this campaign for when you have a more attentive party - especially one where you can tell the players what they're in for to make characters in reaction to it. This is my least favorite idea but if they have these in depth characters that run parallel to the world you're running, it may be worth considering.
Motivation Issues
I'm hoping these are not the CN players who just enjoy what I call the Video Game Experience. To me, the Video Game Experience is that to the player, anything in the environment they can interact with needs to be important, monsters are just experience, and the plot itself doesn't necessarily need to matter if they get the levels and the loot. In this case they are almost turning themselves into NPCs if their actions really have nothing to do with the party.
It may be worthwhile to sit down with them to explain that these characters need a few tweaks to make the game harmonic. While a character can be selfish, the fact is that it's hard to run a full party with the truly self indulgent within it. Tell them that they can keep their characters, perhaps even as they are but the player becomes responsible at least up front with why the character is with the group. After encountering a situations where the characters don't mesh or the lone wolf amidst the team, DMs in my area have started telling people that even if they make characters on their own, it's up to them to make party cohesion work. Sometimes it doesn't matter and the DM holds the leash for some major plot hook, but the fact is that in the seemingly sandbox world you have, the party dynamic is player driven.
So, how do I get out of the vicious circle?
Stop doing the thing that's causing it. You diagnosed this yourself:
It's probably the worst issue I have as a Game Master, I think of a Game, I write a campaign plot for it, End, Beggining and Middle, get Hyped, Hype my players, and after 2 months I want the story to end, and it's usually too late to make changes to the GMing style I use by then, so I just dissapear some weeks and invent I have stuff to do, cancell the game, and run another one.
- You find a game, like D&D 4e, and a beginning-to-end plot for it.
- You play it for a couple of months.
- You cease to be interested.
- You find a new and shiny thing to be excited about—which is fine, there's lots of new and shiny things it's pretty reasonable to be excited about. This might be another game like D&D 5e, or a new kind of story.
- You lie to your players (that is what you're doing) and stop the current game.
- Return to step #1, and repeat.
It's absolutely no surprise one of your players seems concerned: she is concerned, and has good reason to be. It sounds like you've done this at least a couple of times, and your players probably realise by now what's going on—including that you're not being honest with them. You are clearly not having trouble running games in general, though you have stuff to do sometimes. That's fine! Strangely enough, though, whenever you're available to GM again, you aren't interested in continuing the previous thing.
It doesn't take much for someone to see through to what's really going on. They have probably recognised you're just becoming disinterested in stories. Now it has reached the point where you're beginning to hear concerns about it.
That said, it takes some courage to confront this stuff and ask people about it, so kudos to you for having that courage and doing that.
So, how do you break your cycle and stick with one game and enjoy it?
There are always going to be new and shiny things. You are totally justified in getting excited about them. The problem is not that there are new and shiny things, it's that you're letting yourself cancel your games all the time prematurely.
Cycles get broken by recognising the steps, and doing something that interrupts the normal progression through them. Change a step or a critical detail in it, or do something that blows everything apart.
There's a few options to break this problem:
1. Run shorter campaigns
Run shorter stories that only last a few weeks. Have player buy-in that they are going to be short campaigns and that the story will not necessarily continue. (Of course, nothing's stopping you from continuing the story a few months later.)
This lets you switch every now and then harmlessly.
2. Keep it fun for yourself too: don't plan the middle and end.
You probably bore yourself out during your games because you already know the entire story. For the players it's all new and exciting. You already had all that excitement while you were writing the story, and now you're going through the motions waiting for all the stuff you planned to happen, which is boring.
I suggest you stop writing stories that have a beginning, middle and end. Design stories that have a beginning, and then put your players in that beginning, and discover the story with them. This means you're having excitement as well as your players lead you in unexpected directions, and your planning & development fun is spread out over the full length of the game.
BESW and I run games like this: generally, at the beginning, there is a situation at a tipping point where the status quo is about to be disrupted (e.g. two factions are reaching the peak of tension and about to break out into war, and a third faction is rising in the shadows), and the players are positioned as the perfect X factor, with the power and autonomy to influence how events pan out. And then we let them pan out, and give the players and their characters that autonomy. We loosely plan only a session in advance, and we have people with their own individual goals (and not scripts) who we can improvise easily enough, because the players will probably surprise us by doing nothing we expect, and ruin any plans we might have in the process. And then we have fun discovering the story with them.
Try that out.
3. Have someone else GM
Really. If you think your GMing style is a major problem, have someone else be the GM who is willing to run this in a way you're not. It's pretty simple. Then join in and play with your group.
Also an option: open up and be honest with your players.
Tell them what's going on. Tell them you're having trouble staying interested in the games you're running. Tell them you are getting excited about new stuff. Tell them you'd like to be able to run these stories through to completion, and that you're having difficulty doing so. Admit to them you were lying about having things to do because you weren't sure how better to handle the situation.
This might take a little bit of courage to own up to, but it's going to have a lot of benefit.
This will put this issue out in the open, and open it up for people to talk about it. It will likely only confirm what they already suspect but do not feel they can talk about.
Your players will then actually be able to talk about this openly with you, without so much tension and awkwardness being there. Some of them, possibly the girl who was concerned about the current story, will be able to talk to you about possibilities around revisiting that previous story or character she really liked and wants to see more of.
Some of them may be upset. Some of this upset will be upset they already have there because you keep cancelling games, but have not been able to express because this issue has not been open. Be prepared for that, accept it, and be compassionate toward them. If they're your friends, they're generally going to be on your side.
It will also relieve you of the heavy burden of having to deal with all of this all on your own and having to keep it all secret and everything. It will let you actually talk to them about it, get their help and support, and as a group you'll be able to work out how to go ahead from here, among the options in other answers and this one.
You're only human, so have some compassion for yourself, and let the people who care about you help you through this.
Best Answer
I use graph theory. All you need to do is to have NPCs (and/or places) as nodes and plots as arcs. You can even use something like GraphViz to visualise the graph you created. In general, the more complex the graph, the more potentially complex the plot.
Each link could have a cost associated with it that depends on how hard whatever the arc represent is to find out from the PCs's point of view. Or rather what you think is the difficulty in getting that particular thread. This can be highly subjective and difficult to do as all the information you pass to the players has the same colour! They do not know what is really important and what is not. Then you can do some "shortest path" calculations to see how reachable an end node is from any other nodes.
A limitation of this method is that it is very static -- unless you have a nice little script that you can update for all nodes/arcs as your PCs will do things that the NPCs will need to react to.