You are punished for changing the game time (no matter how much, it seems), but not the system time. Here's what happens:
If you change the time on your 3DS system, Animal Crossing seems to simply think that's the time it is now. Trees grow, events change, the only "bad" things that happen would have happened with the flow of time. Even going back in time via this method seems to work okay, even when going to a day previously played.
If you change the game time, using the option at the main menu of Animal Crossing, you will randomly trip for a while after changing the clock. The duration of tripping seems to last longer than the current play session, possibly the full day? But it eventually goes away. If, however, you were to change the game time this way before every play session you would trip randomly every single session, which is highly frustrating.
Also note that going backwards in time will spoil any turnips you had, since the game knows you went backwards, while it doesn't know you went forwards if you go by the system clock.
tl;dr: Change your 3DS's system time to time travel, do not mess with the in-game clock regularly or will will randomly trip.
This link has some pretty good information on 'Perfect Fruits'. It looks like when you plant a 'Perfect fruit' tree you get a random amount of shakes before it dies. It would be pointless to NOT plant/shake the tree when you have them available. You should always have some perfect fruits on hand for when your 'Perfect Fruit' trees die.
Sidenote from the site: Perfect fruits sold in towns that the fruit is not native nets you 3000 bells (compared to the 600 in your town). So definitely sell your perfect fruits in your friends towns!
Note that you cannot plant perfect fruits of non-native species; if your fruit is orange, you cannot plant perfect peaches; they will instead turn into normal peach trees. However, if you imported a perfect orange into your town where oranges are native, you could plant that orange and it would become a perfect orange tree.
Best Answer
During November there will be a mushroom picking event. During this event (whenever it may be), picked mushrooms can be pieces of the mushroom furniture set.
Other than that, the Perfect Mushroom will cause the player to increase in size for a short period of time. As this effect is purely cosmetic, I'd recommend just selling them (except during the event of course).