When you discard cards in the beginning of the game, where do they go

Are there some common rules on which cards to discard immediately?

That's pretty hard to say because it really depends on your build and playstyle. But the cards that cost 5-10 mana should in most cases be removed without thinking. Unless they're some really powerful high quality cards in which case it could be smart to keep one but only if your other cards seem good.

I think it's not worth risking early game by keeping high cost cards unless you're sure you can pull it off.

Besides the mana cost, which factors should I take into account making this decision?

Spell cards:

Some spell cards are more effective late game, even if they cost low amount of mana. Rather remove them unless you feel like they might come in handy early game. Keep good early game spells, especially those that cost low amount of mana and are able to destroy enemy monsters.

Minion cards:

I'd suggest keeping good taunt cards that are up to 5 mana cost. Taunts are very useful and you never know will you get one when you'll really need him later.

Low cost minions that let you draw more cards in any way should be kept as well. Quite useful.

In the end there's no magical formula which will tell you what to keep since you can't know what you're going to get next by drawing. Sometimes you remove cards you don't need early game just to get them replaced with other un-needed cards.

I did some research of my own instead of waiting for another answer. I played 4 games with my Level 5 Mage (leveling her up to level 8 in the process). In none of the games did I manage to use the whole deck, but I wrote down all the cards that I drew throughout all 4 games.

First, to confirm Resorath's answer, none of the Level 2, 4, and 6 cards of the Mage ever showed up in the deck, but the original 5 cards that the Mage starts with all showed up in at least 3 games out of 4. So I can conclude that none of the later Mage cards are included into the basic deck as you level up.

Second, exactly ten other cards showed up in 2 to 4 games each out of 4, while no other cards ever showed up. So, only 15 different cards ever showed up for the Mage, some twice, making up the deck of 30. That means, it's not random, and the cards are set in for each basic deck. Most likely, they are tailored to each class. (For the curious, for Mage they are: Murloc Raider, Bloodfen Raptor, Novice Engineer, River Crocolisk, Raid Leader, Wolfrider, Oasis Snapjaw, Sen'jin Shieldmasta, Nightblade, and Boulderfist Ogre, plus the original 5 Mage cards)

Both of those conclusions could be very good coincidences, but, eyeballing the probabilities of these coincidences, I firmly believe that those conclusions are correct.

So, I did not experiment with all 9 classes, but I found my answer: The basic decks were pre-made by the creators of the game, and they do not change as you level up. I guess the results for the other 8 can be found the same way.