You have the right answer in your choices.
Being a DM isn't about writing a script and continually nullifying player choices to keep them "on script". If you want to write a story without much outside input, then write fiction. Nothing wrong with that.
A DM is only one participant of the story when role-playing. Sure, typically the DM will set up the initial scenario and make certain decisions that will constrain choice. But you have to accept that once you turn the PC's loose on "your" world, they are likely to take your carefully crafted story, and fold/spindle/mutilate it (or all three!).
You can of course plan an overarching story for the campaign. It can a great idea to do so, it provides direction and focuses the campaign. But if the players make the story no longer possible and you can't come up with a satisfying way to "fix" it, then it's time to adjust.
If the party goes off of the rails, it is likely due to one of three things
- They notice the rails and want off! Whether by boredom or active malice they've decided to go left instead of right. Either way, you're getting feedback that your story is not as entertaining to them as you might have thought. Time to make changes!
- Player discovery - The players have discovered something cool about your campaign or their characters that you might not have thought about and want to explore it! This is awesome, it will mean that your story should go on the back burner for a bit. When the players show an active interest in the setting, nurture that, don't shut them down. Let it play out, it usually won't be a long detour and the players will rejoin the main plot-line with renewed vigor.
- Player Agency - One or more players has their own story that they want to explore. Quite often a player will build a backstory for their character that has some hooks in it. If you don't grab the hooks and work with them (I think we're all guilty of this at some point!) the player may still want to explore their half-dragon ancestry, or why they got kicked out of Star Fleet Academy. Let them run with it, rein them back in if it becomes excessive. In the same way that your story cannot dominate the table you can't allow one player to continually dictate what happens next. However there should always be room for each player to shine for a session now and again.
The last two are sides of the same coin. Both involve letting players "run the show" to a certain extent. One problem that I've seen multiple times is for the GM to plan one scenario after another after another, never giving the players a chance to catch their breath. Instead of plotting things so extensively, ease up occasionally. Let the players know that after the current big boss is killed that there are no immediate plans for the campaign and ask, "What will you want to do?" This can serve as a release valve and free up any building tensions.
Good question. There's usually a deeper motivation behind going out and stealing things. Your characters will probably fit into one of the following three buckets and this will give them a reason to start.
Greed
A greedy character often wants to use money as a way to keep score. Thievery is an easy way to get it AND bring somebody else down a level.
This can be considered the default for most D&D style rogues.
Mastery
Somebody who is good at a skill will often want to improve it. They could have started out being trained by older mentors (small hands and an innocent look make for great pickpockets) or stole for survival. After more and more use, this then becomes the only skill that they think they can use.
At any moment, a chance to use and improve those skills becomes something that someone will try to take. The bigger a risk, the bigger the thrill when you succeed.
Survival
This is usually the way beginning rogues get involved in "the business". It's better than starving and there aren't any other chances. Once you've started stealing, it's easier and easier to justify doing it just one more time.
If anybody knows, blackmailing the character to do some more dirty work gets easier. Each time they break the law they get deeper and deeper. This is often how dirty cops in fiction get buried.
How To Use The Motivations
The motivations themselves can help you get players onto the edges of a big quest.
Telling a greedy character about a legendary dagger that is rumoured to be able to kill the gods is a sure fire way to get him looking for it. Make the clue-chain reasonable and he will beeline straight for it.
Tell a survival oriented character that he and his family will be well taken care of. They just need to do another little job first... oh, and another. Because we have your darling little kids.
A character that wants mastery will generally be intrigued by a challenge. NPCs complaining that the security in a particular building is too tight and they can't see a way in should be enough to perk his interest. Have him find something interesting that hints at a bigger challenge somewhere else and he'll be raring to go.
Beyond The Motivations
The most effective way to make players focus on a Big Bad is for it to threaten them specifically. Well, not exactly them, but their way of life and the world around them. There
During the initial phase, whether they end up working for the Mob or going freelance, encourage them to build up ties (and show off any backstory). These are what you are going to threaten.
Have one of the PCs mothers invite him for a visit occasionally. 2 minutes of "She offers you tea and asks after you and the boys you grew up with. What do you tell her?" should be enough to help set the relationship and make her exist in the player's minds.
They might also build up a relationship with a favourite fence, pawn shop owner or barman. They could take ownership of a building as a hiding-place, to cook drugs or run a brothel in. Whatever they feel like.
The purpose of this phase is to let them build up bits of the world that are important to them. It's important that they choose these, so you'll need quite a few pieces of colour that you can expand on as their interactions deepen.
Later on, you start to use the ties to build tension. For example, have the mother talk to the PC about strange men prowling around (and there's nobody there when he checks) a couple of times. Tell them they can't find their fence, that nobody's seen him all day and that his place was tossed.
Once they find out who their enemies are, you can bet your ass they are going to go out there and deal with them.
Summary
At the end of the day, to go beyond petty theft and feathering the nest, it's all about people. Who they like, who rubs them the wrong way, who seems to have a similar agenda, who can get them owing (and paying) favors.
Best Answer
You're in quite a difficult situation. Your players don't have information, don't have many leads, have one dead party member, and have been launched into confusion.
Slow down the overarching plot of your game - grind it to a halt for now, if you need to. Your players (and their characters) both are not ready for it and do not have the information they need to process it. The way this game is being played, you have significant control over the plot and pacing of the game. Slow, possibly to a halt, the destruction of the universe and the whole ancient magicks and dragons shindig. Your players know it's an imminent threat, but lack the skills or knowledge they need to confront it in any way - they need time to learn and grow.
In a lot of ways, this can serve simply to set the scene for whatever else is going on, and contextualize the additional information the players gain from other goals and quests. Let it do so - your players will learn about the overall problem, and potentially be able to connect the dots where others have not. At a later time, once the players have better information, they'll be able to handle the overarching plot. Until then...
Give your players information. Consistently, the examples you've communicated have had a common problem: the players achieve what they in the scene want, but don't get any information about what they want in the broader scope. There are two effects from this: first, the players will grow frustrated; second: the players will quickly run out of leads. The leads they get need to go somewhere.
Maybe occasionally you can throw in some lead that just doesn't work, but the majority of them should provide some sense of success. The players learn something, or gain something. They don't have to completely understand it, but something needs to happen. Otherwise, all the plot hooks will drop. For instance, while interrogating the cultists, some design element of this scene needs to provide the characters with more information about who and what they want to be pursuing.
Always ask yourself, "Would the players really enjoy this right now?" - and give a more detailed answer than just yes or no. For instance, concerning the meteor falling, ask yourself, "Would the players enjoy the meteor falling right now?" The answer to this question is, as you've retrospectively identified, something along the lines of "No, because the players do not have the information or skill they need to handle the situation."
You've got a good scene - meteor falls, chaos ensues - but the players aren't prepared to handle it in a way that makes the game fun for them. If the game were run again, this scene should be saved for a time when the players can deal with it in a meaningful way. By answering the "is it fun?" question, you've learned two areas which must be fulfilled before you can safely play out this scene: the players need to be more powerful, and the players need more information. Once the players have achieved this, then you can go and run this scene.
Obviously, this scene has already been run, and I'm only using it as an example. Short of rolling the game back to before this scene (which may be a completely reasonable course of action, depending), this is part of the game world right now, and is just something the players are going to have to deal with.
This is the core of my advice. But how does it apply to your campaign?
Here's what I'd strongly advise: set aside things the players can't handle for now. They can evolve in the background, but as a general rule, they shouldn't affect the players until they're ready to be drawn into those plot threads.
Come up with some scenes that your players would all want to play, and would all be able to handle, and draw the characters into them. Your players will decide what they want to do next, and it's probably going to be to explore deeper. Through these scenes, you can feed them pieces of information, ideas, thoughts, and leads, which will draw them both a) into more power through the experience system, and b) into greater understanding of the forces at work in your game.