And the very next evening I solved the problem.
Once you rescue the hermit, he will offer to train you. By beating him over and over (he isn't fighting with his real power, so you won't take damage and beat him and his fireballs with a single hit), you can gain the levels needed to beat the stage.
There are also two apprentices of the hermit hidden in the stage. Each will give you one of the hermits favourite foods and giving those to the hermit will make him stronger, thus allowing you to level up faster.
However, if and only if (this is what I failed to fully explore) you give him both of these food items, he will stop training with you at level 40. If, however you give him only one of the foods or neither, you can keep leveling beyond this, up to around level 60 where you'll start to run out of money to buy more time.
What I found was that giving him neither foods slowed the leveling up process so much, you didn't have time to get to level 60 before running out of money. But by giving him one of the foods, training untill I was level 60 and then (finding and) giving him the second food item, he would stop training me, challenge me and I was able to beat him.
And for completeness sake, my gear was Crimson Robe, Horned Helm, Spike Mace, Flash Shield and Bigfoot.
The shield you buy in the southmost town is not optional and skipping it wastes more time than it saves. As the game says at some point, shields are necessary to absorb critical hits; in this particular level this means reducing the amount of knockback you get from the final boss.
Dash to the south town past the bridge, buy herbs for the fight if you want and buy the shield. Dash to the castle and you'll make it (more or less) barely in time.
Best Answer
In this level, the forests just south of the east-west mountain range are shaped like an NES controller, as the picture shows.
To access the secret, enter the code
by walking on to each corresponding space in order, and without walking onto any other code space in between. Each code space can only be activated if you step on it after it turns sepia. Each step that correctly activates a code space will be accompanied by a tone and the game will pause temporarily, and each incorrect step will be accompanied by a different tone and the code will reset. The following path will correctly input the code:
Once the code is properly input, a cave will appear at the center of the + shape of trees and on the other side is a town where a girl will give you glasses. Be careful because if you reset time after you enter the code, the cave will disappear and the code can not be reentered.