Grammatically, "a perfect start of the day" makes sense. And without the word "perfect", it would usually be the best choice, as in "at the start of the day".
But there is an established idiom for phrases like "a good start to" when describing some period of time or event, where the adjective good can be replaced with many other adjectives describing value quality: great, perfect, excellent, bad, poor, disappointing, etc.
Since probably most English speakers are used to this idiom, it will be better to say "a perfect start to the day".
(Interestingly, it looks a century or more ago, "a good start of" was more common than "a good start to", but "a good start to" has been steadily becoming more frequent: see the Google ngram comparison.)
Best Answer
You could consider "the next day" as a adverbial phrase, so the sentence
is grammatical and idiomatic.
Another example from Collins dictionary could verify this kind of use.