You enter turn-based action when you need to track time* that closely. You leave it as soon as you don't need that close tracking.
Turn-based time tracking dominates combat, but it can also be used for chases (DMG 252), complex traps (DMG 121), even tense social scenes. That said, it's a really clunky mechanism, right up there with alignment and "what HPs mean" for likelihood of starting an argument.
You're right: while the books say that initiative is rolled at the start of every combat encounter**, they say nothing about when to leave initiative. Common sense has to be our guide: we clearly don't want to always be in "combat-time," or we'd never get from one side of town to the other. Given that we will sometimes not be in combat-time, the question is then: when? Whenever you don't need it--initiative, and combat turns/rounds, are a tool for you to use or not as you need.
A few observations, from my experience playing and running:
- initiative variants can also play strangely with "roll for initiative"-based features--be thoughtful when employing them;
- I don't even use initiative for every combat encounter--sometimes it's not worth the trouble (in my opinion) and I'll suggest to the table that we run a simple little simultaneous-declaration-and-resolution combat;
- escaping combat while in turn-order is largely broken in 5e, in my opinion--the moment either side decides that "running for my life" is more important than "exerting my will by force of arms and might of magic" I break out of combat spacetime.
* - To the extent that initiative and turns/rounds even do that.
** - Which can be a little clunky, like a voice from the gods screaming "MURDER NOW!" To fix this I ask players for a set of initiative rolls each time we leave turn-order, to have on hand the next time it arises.
Rule 0
On one hand, the DM can do whatever they want; they are the DM. But, by the actual rules in the book...
No, this is not how the rules work as written
The description of Initiative states:
At the beginning of every combat, you roll initiative by making a Dexterity check.
When the creatures are "readying an attack", this is already part of the combat; therefore, Initiative should have been rolled, and you would not be surprised by the attack. This interpretation is backed up in this similar Q&A.
Note that the creatures are still hidden (if you didn't perceive them), so, even though you are not surprised by the combat, not necessarily you know where the creatures are. Even if they are hidden, the Alert feat also states (PHB, p. 165):
Other creatures don’t gain advantage on attack rolls against you as a result of being unseen by you.
By the way, in your text, you mention that the swarm of insects
immediately attacked him, dealing [x] damage.
From this, it is unclear whether the DM rolled an attack roll as well. Even if surprised and attacked by a hidden creature, the creature still has to hit with the attack roll before dealing damage.
Talk to your DM
It seems this situation has upset you. You have taken a feat, giving up an ASI, with the sole objective of such situations not happening, and then the DM pulls up something not entirely backed by the rules that makes the entire point of taking such a feat get thrown away.
Talk to your DM. Tell them that how they handled this situation was upsetting. Explain that the point of taking such a feat is to avoid these situations, and that by house ruling otherwise, they are harming your character's concept. At worst, ask them to let you change the feat, if they are going to keep doing such things, because your expectations when building the character were different.
Best Answer
There is a combat variant in the 5e DMG, page 270, "Side Initiative" (emphasis mine)