Fresh Ground Coffee vs Coffee Grinder – Importance Comparison

I've owned several grinders -- both blade and burr, but I've never had one that did this. So no I would not say it is normal

Is the lid closed tight when you start it? Blade style ones are pretty cheap, so you might just want to get a new one.

Posted as answer by request of @BaffledCook:

Here's an slightly informal blog post outlining changes in taste between varying degrees of freshness in the grind of a coffee.

http://investigationsblog.wordpress.com/2010/03/03/do-i-really-have-to-grind-coffee-right-before-brewing-it/

The short version is that the coffee starts losing freshness as soon as it is ROASTED. The longer it sits the faster it goes stale. The more surface area it has (ie: you've ground it up; also, the finer the grind), the faster it goes stale. The more you expose it to air, the faster it goes stale. So, seal your coffee in an air tight container at room temperature. And grind as close to the time you add water to it as possible.

How much of a difference detected depends on the the actual coffee been, the roast, the taster's taste buds. If you buy a bean that's been sitting on the shelf for 3 months already, you will probably notice less of a difference than a bean that was roasted last week. I buy beans that are roasted and sold within a week, and I DO notice a difference if I leave the grounds for a day or two before drinking.