First and foremost, a GM should always remember that the objective of a game is to have fun. The thing is that "fun" can mean different things to different people, and it sounds like what's fun for you to create isn't as fun for the players when it's executed - and the end result isn't all that fun for you either, since you're sharing this issue here.
Here are my suggestions:
- Create expectations for the game world before you begin play. By talking to your players and letting them know you will not be pulling punches nor delivering "level-appropriate encounters" exclusively, they are more likely to on their toes and less likely to rush into danger because they (as players) KNOW that there is a possibility that this particular adversary is too strong for them. If this happens, though, expect to see a decreased reliance on combat resolution on behalf of the PCs.
- Treat the players actions with the same degree of respect you expect from them. A player can only control his or her character in the entire game world. If they do something that should trigger a response, make sure your NPCs respond in a way that is congruent with the setting and with the knowledge that the NPC have of the actions (as opposed to the knowledge that you, as a GM, know). For instance, if your players kill off a guardsman but do it without leaving obvious traces, consider how far the authorities would be willing to go to track down the killer and respond accordingly. Ever played a videogame in which you kill off an NPC in a secluded area of the game world and cops/guards immediately show up guns blazing even when no one SHOULD even know a crime was committed? If your players feel like that, they have no reason to be careful, since they feel the game world will just react according to the GM's whims.
- Danger is fun; immutable doom is not. As a GM, you always have the power to kill your players, but there are ways that are fun and ways that aren't. Since the job of a GM is partly to provide entertainment (for himself AND the other players), make it a point to find ways to throw threats at your players without instantly killing them. "Lightning strikes you. You die" is a perfectly legitimate thing for a GM to say. After all, he controls the world, which includes weather. Is that fun for a player, though? The example with the assassin that you mentioned, from the player's perspective, amounts to pretty much the same thing. As cool as the concept of the undetectable assassin is, the assassin missing his first shot (or perhaps not scoring a killing blow) makes for a gaming experience as the PCs scramble (they don't know if more shots are coming their way), try to find out who fired the shot (and probably fail since your assassin has escape plans), and are generally spooked because the situation just demonstrated that a) someone is out to kill them, and b) they're good enough to get away with it undetected. This puts the ball in the players court, as they decide what to do about it, and lo and behold, you have a game.
As a final comment, I'd like to recommend Robin's Laws of Good Game Mastering. This book has plenty of tips and hints on how to make your sessions fun for both yourself and your players. It's a resource you might find useful to liven things up around your gaming table.
So first of all, I really like this idea. I tried it once or twice a long time ago and I'm still fantasizing about it. With that out of the way, let's move to my 2 cents.
Make the NPCs well-rounded characters
These NPCs aren't just some recurring characters, not to mention some one-time ones. They're gonna be with the party for a very long time. Whole adventures, actually, which is quite a lot. This means that they should be really well rounded. Your players should get to the point of distinguishing between them after just a few words, especially when the NPCs are the ones who spark the conversation.
More than that, though, they should be more than cardstock characters. You want them to express feelings, you want them to have goals, and you want them to react. Otherwise, your players will have hard time connecting with them, and this is far less good. As a rule of thumb, make them round and distinguishable from each other. A wizard is different from a fighter, true, but we want a greater difference. The prince will have hard time adjusting to the wilderness and to the bugs in his bed. The merchant will always stop to collect the better loot so she'll be able to earn much from those bargains. Classes are a dirty way to distinguish, but they're better when accompanied by some other traits.
Don't let them speak right one after the other
The players aren't coming for the game as an audience for a theater, they're here to play, and hearing NPCs talk with each other is far less fun. If you must make them speak between themselves, make it as quickly as you can and immediately move on. If you can narrate the conversation instead ("they're talking about what happened, Elsa thinks that they should go east and Hans thinks that they should go west…"), it is far far better.
Give the players the center stage
The players and their characters are the real stars of the campaign. The NPCs are extras, and should always be seen like for you that when you're playing them. If they're far cooler than the PCs, or if they have much more screen time, something is off with your campaign. It's better to not have the NPCs with the group than to let them illuminate and shadow the PCs.
They're not all knowing
Think for yourself what is more important, a character who knows everything or a character who knows only part of them. For me, it is the latter. That comes from one simple thing: There's no drama when everyone knows everything (it is not entirely true, but that's for another time). Make them say sometimes stupid or idiotic things, make them come to wrong conclusions, let them make mistakes. They're not a kind of supernatural deity who knows everything, but humanoids who are as humane as the characters, and they should be played this way.
Make them important for the story
They should always be important to the overall story, in one way or another. In one of my more successful D&D campaigns, the characters had to escort a princess to a neighboring kingdom through the forests. Having to keep her safe from one hand, and dealing with all of her complaints from the other one made the game so much richer. They don't have to be important to each and every one of the scenes, but they should always be important for the overall story. Otherwise, the characters may just leave them to rot one day, when things will turn the wrong way.
And an end
Hope I succeeded with helping you a little bit.
Best Answer
Make them stumble upon and want to bite the hook.
Expecially in a homebrew campaign, players will want to get their bearings: take a look around, see what the locals are like etc. This is where you come in:
Now, you do not need to have every event from killing dragons to going to the outhouse involve a plot-important event. Instead, let them use the village where they start as their sort-of home base, where they get invested in by doing things big and small. But here's the twist: after every second or third sidemission something happens regarding to the plot hooks: wounded travelers come in, speaking of bandits; one of the local lord's men is in search for people who could lend their swords to the aid of the realm, or have Sir Murder von Evilstabbings ride into town, off some folks, then go on. Eventually this will drive the players to want to take the hook, if only to put an end to the schemes of Larrenyyys the lich.
Have their actions improve the game world, then have setbacks (minor or major) kick in until the party kicks someone's door down and give them greatsword accupuncture.