Card Counts for Expert and Goblins Versus Gnomes Packs

Are there some common rules on which cards to discard immediately?

That's pretty hard to say because it really depends on your build and playstyle. But the cards that cost 5-10 mana should in most cases be removed without thinking. Unless they're some really powerful high quality cards in which case it could be smart to keep one but only if your other cards seem good.

I think it's not worth risking early game by keeping high cost cards unless you're sure you can pull it off.

Besides the mana cost, which factors should I take into account making this decision?

Spell cards:

Some spell cards are more effective late game, even if they cost low amount of mana. Rather remove them unless you feel like they might come in handy early game. Keep good early game spells, especially those that cost low amount of mana and are able to destroy enemy monsters.

Minion cards:

I'd suggest keeping good taunt cards that are up to 5 mana cost. Taunts are very useful and you never know will you get one when you'll really need him later.

Low cost minions that let you draw more cards in any way should be kept as well. Quite useful.

In the end there's no magical formula which will tell you what to keep since you can't know what you're going to get next by drawing. Sometimes you remove cards you don't need early game just to get them replaced with other un-needed cards.

An answer to your question depends on what you are trying to get out of the game.

Collecting all of the Cards!

According to the Economy of Hearthstone, in order to collect all of the cards, you should save all of your dust until you have enough to craft the cards you have missing. This article simulates optimal dust usage to calculate the total number of packs you would need on average to collect all of the cards.

Creating the Best Deck

Creating an excellent deck does not require a lot of dust or legendary cards. Trump was able to get to the Legendary rank very quickly without spending any money--you can see it here. In this series he spends his dust on several commons, a few rares, and a single epic (if I remember correctly). Most of the top players have a few legendaries in their decks, but it is not required. Decide on a deck you want to create and spend your dust getting those cards. The optimal way to spend dust to create a particular deck as fast as you can is to save up your dust until you can craft the rest of the deck.

Creating a Decent Ranked Deck

Perhaps you just want to play ranked and "rank-up" relatively quickly. The best way to do this is not to attempt to build the best deck which lots of rare cards, but to build a good deck with strong cards.

1. First, decide on a deck strategy that relies mostly on rares and commons. 90% or so of your deck will be made of these, so they have to work on their own with support from epics and legendaries.
2. Second, fill out the deck with cards that you already have that would work well in your dream deck.
3. Third, fill the rest of the deck with cards you already have that may not be your ideal picks, but are at least good.
4. Fourth, play with your not ideal deck. For optimal dust usage, you want to determine what is missing from your deck and this is best done through experience rather than theory-crafting.
5. Spend your dust on the cards your play experience dictates would be the best additions. If you can't afford all of your ideal cards, spend your dust getting the largest number of ideal cards you can. This is an optimal use of your dust because it gets your deck as close as possible to your ideal deck.