Here's a technique I've used. When I invite people to a game I tell them that the game we're playing is a homebrew system called "Valadil's Game" which is loosely based on D&D.
This does a couple things. Firstly, it scares off rules lawyers who want to play RAW. I figure those players aren't compatible with my games anyway and I'd rather just nip that in the bud. It also signals to the players that this isn't another kick down the door, slay the monster, loot the treasure D&D game. It removes that expectation and opens them up to something with more story.
If you want to get technical about it, this is just a restatement of Rule 0. But it works.
Every session offers a chance for veto. Demonstrate central mechanics first, supporting mechanics (like character gen) second.
While it's nice, in theory, to say "We'll try this for 6 sessions" it is quite clear, to me, within one session of playing a game if I'm comfortable with the game or not. Just about a year ago, I did a number of games in "uncampaign" format, where there was a weekly meeting to try a different game as often as we felt like. Games the group didn't like didn't last beyond one session because it was obvious that the mechanics didn't mesh well with group expectations.
The trick is to keep things simple. The first session is to explore and demonstrate the core mechanic of the system. The central mode of conflict should be made ovious in the first session. If characters are sufficiently complex that they take non-trivial amounts of time to generate, they should be presented beforehand.
In running professional demos of the RPG I helped write, we had to communicate the central concept in a five minute combat. A four hour game was enough for a whirlwind tour of all the major mechanics and the length of time most convention games are budgeted for.
My advice: Run one game as a pre-gen demo game, and allow players to create characters during the second game of the system demonstration. Allow anyone to veto for any reason, and have a debrief after to see what went wrong for people. People are, of course, allowed to make persuasive cases and explain the difference interpretations of rules in the debrief, but it will bcome quite clear if everyone is willing to try a second session.
Best Answer
For D20/D&D3.5 - things like grappling/sundering/size disparity/jumping I often get it wrong.. so I would often rely a lot on the collective mindshare of the other players. If I come to a point that I can't remembrance, I usually offer something like "I seem to remember that it works this way.." give my solution and if nobody objects, well, that's the way it works. So the first tier is "Proposed DM solution"- You have to make it clear that if your way is wrong, you'd be open to correction.
Usually/often I have had someone (sometimes two or three people) who was way more into the way the rules worked (perhaps because they had built charaters around those rules) and they would clarify the procedure before the dice roll. The second tier is "How Joe the expert is pretty sure it works"
Checking the rulebook at that point becomes the third option - only if everyone agrees that they don't know how it works, but that my proposed solution was wrong, and that the situation is serious enough that it warrants a rule vice a ruling.
I always hope never have to get to third tier.