You are punished for changing the game time (no matter how much, it seems), but not the system time. Here's what happens:
If you change the time on your 3DS system, Animal Crossing seems to simply think that's the time it is now. Trees grow, events change, the only "bad" things that happen would have happened with the flow of time. Even going back in time via this method seems to work okay, even when going to a day previously played.
If you change the game time, using the option at the main menu of Animal Crossing, you will randomly trip for a while after changing the clock. The duration of tripping seems to last longer than the current play session, possibly the full day? But it eventually goes away. If, however, you were to change the game time this way before every play session you would trip randomly every single session, which is highly frustrating.
Also note that going backwards in time will spoil any turnips you had, since the game knows you went backwards, while it doesn't know you went forwards if you go by the system clock.
tl;dr: Change your 3DS's system time to time travel, do not mess with the in-game clock regularly or will will randomly trip.
You do not have to pay the loan off all at once, and infact the loans will quickly exceed what you are even capable of holding on your person out of item form very quickly (You are capped at around 100,000 bells before you have to put them into the bank machine or bags of bells).
Best Answer
I don't think you're doing anything wrong. It should take 3 days. In any case, from the date you've posted this, I can only assume that it completed in all this time!