It's almost correct, but not quite.

There are two separate, unrelated ways to get a familiar in D&D 5e, and you're trying to combine them.

The first method is by use of the Find Familiar spell, which allows you to summon a celestial, fey, or fiendish spirit that takes the form of any of a list of creatures. This list is expanded by the warlock pact of the chain. This familiar is perfectly obedient, can be resummoned when it dies, can be hid in a pocket dimension, deliver touch spells that you cast, and everything else specified in the spell description. This method gives you a familiar with the basic stats of the chosen creature, not the "variant: familiar" traits of the chosen creature (unless your DM chooses to have that creature type appear).

The second method is by finding a quasit, imp, or pseudodragon that has the "variant: familiar" trait (which is 100% up to the DM), and enlisting it as a familiar by interacting with it. This familiar has only the traits listed in the stat block for that creature, including the variant traits, but none of the traits of familiars given by the Find Familiar spell. No pocket dimension, no touch spells, no limitations on what actions it can perform, and if it dies, it's just dead. This type of familiar is an NPC controlled by the DM (much like a hireling or other follower), and is only as obedient to the PC as the DM says it is, using the MM entry as a guide.

Now that we've established how things actually work, we can address your real concern. Your warlock can't yet communicate at a great distance, but he can soon. There is a warlock invocation available to him called Voice of the Chain Master that does the same thing, but with unlimited range on the same plane. As you've realized, this ability has some incredible potential, especially for scouting.

It's not any more powerful than other options and should not be limited.

Let's compare it to some other invocations. There's one that lets a warlock cast Disguise Self as at will. This would let him see a guard, and appear exactly like that guard and just walk around the enemy camp unimpeded. Or maybe impersonate the leader of the camp and just take it over without even a struggle. There's another invocation that lets the warlock cast arcane eye at will, which gives you a way better scout than an easily killed creature. A familiar, even an invisible one, still has to succeed on a Dexterity (stealth) check to avoid being heard and then easily killed. An arcane eye does not.

So, in order to fully utilize this scouting ability, your warlock has to pick the chain pact and spend one of his few, precious invocations, both of which are huge opportunity costs. He deserves something in return. This something is you not limiting it. It already has a flaw in still being able to be killed by anything that hears it, or smells it. That's right, just about any pet wolf is going to be enough to catch this familiar. It doesn't need any more limitations.

Channeling a spell through a familiar ends their invisibility

As you've already determined, the line about "...delivering the spell as if it had cast it" is enough to trigger invisibility ending on a spell cast.

Any of them would lose their invisibility after delivering a spell

Quasits and Imps can't normally cast spells so their invisibility blocks don't list "or casts a spell" as an end condition. Since all the invisibility stat blocks do list all of the offensive actions the monsters are normally capable of as ending invisibility it's pretty easy to figure out what should happen here. So all of these monsters will end their invisibility if you channel a spell through them.

A strict RAW ruling would disagree, but 5e just isn't written tightly enough to support that kind of ruling.