It's almost correct, but not quite.
There are two separate, unrelated ways to get a familiar in D&D 5e, and you're trying to combine them.
The first method is by use of the Find Familiar spell, which allows you to summon a celestial, fey, or fiendish spirit that takes the form of any of a list of creatures. This list is expanded by the warlock pact of the chain. This familiar is perfectly obedient, can be resummoned when it dies, can be hid in a pocket dimension, deliver touch spells that you cast, and everything else specified in the spell description. This method gives you a familiar with the basic stats of the chosen creature, not the "variant: familiar" traits of the chosen creature (unless your DM chooses to have that creature type appear).
The second method is by finding a quasit, imp, or pseudodragon that has the "variant: familiar" trait (which is 100% up to the DM), and enlisting it as a familiar by interacting with it. This familiar has only the traits listed in the stat block for that creature, including the variant traits, but none of the traits of familiars given by the Find Familiar spell. No pocket dimension, no touch spells, no limitations on what actions it can perform, and if it dies, it's just dead. This type of familiar is an NPC controlled by the DM (much like a hireling or other follower), and is only as obedient to the PC as the DM says it is, using the MM entry as a guide.
Now that we've established how things actually work, we can address your real concern. Your warlock can't yet communicate at a great distance, but he can soon. There is a warlock invocation available to him called Voice of the Chain Master that does the same thing, but with unlimited range on the same plane. As you've realized, this ability has some incredible potential, especially for scouting.
It's not any more powerful than other options and should not be limited.
Let's compare it to some other invocations. There's one that lets a warlock cast Disguise Self as at will. This would let him see a guard, and appear exactly like that guard and just walk around the enemy camp unimpeded. Or maybe impersonate the leader of the camp and just take it over without even a struggle. There's another invocation that lets the warlock cast arcane eye at will, which gives you a way better scout than an easily killed creature. A familiar, even an invisible one, still has to succeed on a Dexterity (stealth) check to avoid being heard and then easily killed. An arcane eye does not.
So, in order to fully utilize this scouting ability, your warlock has to pick the chain pact and spend one of his few, precious invocations, both of which are huge opportunity costs. He deserves something in return. This something is you not limiting it. It already has a flaw in still being able to be killed by anything that hears it, or smells it. That's right, just about any pet wolf is going to be enough to catch this familiar. It doesn't need any more limitations.
Channeling a spell through a familiar ends their invisibility
As you've already determined, the line about "...delivering the spell as if it had cast it" is enough to trigger invisibility ending on a spell cast.
Any of them would lose their invisibility after delivering a spell
Quasits and Imps can't normally cast spells so their invisibility blocks don't list "or casts a spell" as an end condition. Since all the invisibility stat blocks do list all of the offensive actions the monsters are normally capable of as ending invisibility it's pretty easy to figure out what should happen here. So all of these monsters will end their invisibility if you channel a spell through them.
A strict RAW ruling would disagree, but 5e just isn't written tightly enough to support that kind of ruling.
Best Answer
You lose the ability to cast find familiar when you change pacts.
Pact of the Chain says:
So Pact of the Chain is the source of find familiar. If you switch from Pact of the Chain to Pact of the Tome, you lose the ability to cast find familiar familiar. So the question "what happens when I recast it after changing pacts" is moot since you cannot cast it at all.
Your familiar will hang around after you switch pacts, but once it drops to 0 hit points or you "dismiss it forever" it is gone for good, unless you somehow gain the ability to cast the spell again. However, it should be noted that it is entirely reasonable for a DM to rule that the services provided by the familiar are part-and-parcel to your current pact agreement, and changing terms would cause your familiar to vanish.
If you later find a way to recast find familiar, you may only choose one of the usual options, not the options provided by Pact of the Chain.
This part is quite simple: Pact of the Chain is what gives you the ability to choose one of the special forms. Without Pact of the Chain, you may not select one of the special forms. You always select from the available forms when you cast the spell, since the spell says:
The phrase "it reappears after you cast this spell again" is simply indicating that you must cast the spell again to reacquire the services of your familiar - it isn't bypassing the rest of the spell description.