Yes, that was fine.
You're the DM: you're there to challenge them and put them in danger, not to keep them all safe.
A lot of this comes down to the kind of game you all play. Statements like "dice rolls should never kill a character" or "you should never have hidden enemies" are total BS. Those are valid agreed-upon social contract items for your group, but they are not generally true statements. Players in my group would recoil in horror at the thought of having either of those rules in place.
Now, some of the problem may be a lack of common understanding of the kind of game you are all in, and it's worth a discussion about what you consider to be the parameters of your campaign - how can characters die, what kind of tactics are fair, etc.
Hidden Casters
This is fine. PCs love to put on Invisibility and cast spells, bad guys can too. I just ran a game two weeks ago where there were melee opponents and then an invisible summoner who was doing a lot of stuff. The PCs figured that spell effects don't just come from nowhere, made Perception checks, cast Invisibility Purge, bing bang boom. Of course, they're smart players... With the familiar, you telegraphed this pretty hard.
Mind Control
OK, players never like mind control, but there's 100 spells that do it and fear you or confuse you or charm you or dominate you. It's a part of the game. Also a valid tactic. Some people claim that this caused the other PC to die with "nothing they could do about it." That's patently false - they have as much as they can do about it as when they get attacked by any other threat (have a higher AC?). There's not always a specific "roll to avoid dying" in D&D. If this was a summoned creature or invisible creature or teleporting-in creature or any number of other things, it would be the exact same lethality.
Maturity
The real problem here is likely one of more of a combination of common understanding of what can happen in the game and maturity. I suspect you're newer gamers and perhaps young. I suspect some of these folks would "flip out" in the same way if they lost a big fight in a MMO or a basketball game, right? That's just general social maturity and what do do about that is out of scope of RPG.SE.
But there's also the issue of newer gamers not knowing all what could happen. If it's the first time they see an invisible caster, then it's quite a shock. Once you've been gaming 20 years that's instead like about the first thing you check for if anything unexplained happens. We have various "if person X gets mind controlled" countermeasures planned out. But for new gamers, the combination of the unexpected with general emotional trouble dealing with loss is a hard combination.
Once you get folks back, it may be worth using this as a teaching opportunity. "Yes, that was hardcore, right? You can do the same thing, be trickier than your opponents! Now roll a new character even more bad ass than the last one!" "Old school" gaming was a continual exercise in this exact thing - requiring you to actually think about what you're doing and not just run forward and grind - and people loved/love it. If they want to play a different kind of game ("I don't ever want to die! I want save points! And nothing should ever hinder my character, mentally or physically!") then you can negotiate the kind of game you want to run and they all want to play.
The essence of sandbox play is following where the players lead, and it sounds like you're already doing that.
What adding randomness does is make the world feel more alive and larger than the thread that the PCs are pursuing/creating, allowing the players to make informed decisions about where they want to drive the game. You don't need to be constantly interrupting them with random encounters, but using them as a "spice" enhances the game and increases the players' knowledge of their environment, which itself increases the players' agency.
And don't worry about diversions derailing the campaign or slowing its advancement—the point of a sandbox is that there is no right path. If your players choose to diverge from their goals… it's because they have more goals than the one. If they want to advance one goal (not "the campaign"), then they will prioritise that goal even when distractions occur.
As long as your randomly-added distractions aren't mini side-railroads that force the players onto them, you can trust that they won't interfere with the players' freedom to choose their direction and speed. If the players choose to be distracted: that is where the campaign is. If they choose to not be distracted: that is where the campaign is.
Just remember that the campaign is where the players lead, no matter whether it was their idea or yours, and you can cut through this Gordian knot of worry about whether you "should" distract them with random things and whether the campaign is advancing at the "right" speed. There is no right speed or direction in a sandbox, only choices and outcomes.
Best Answer
Inciting intrique (without forcing firefights)
Example: If faction A contains the party mage, make sure faction B has some goals requiring identify spells, or detection of magic, or where a fireball would be handy. The intrigue comes from trying to create plausible reasons for the whole party to do what faction B secretly wants it to do.
Ensure that both factions have some goals which simply require the total party strength to accomplish. Set up good reasons to do both sets; make all goals seem like reasonable choices. (For bonus points: then make sure there isn't time to do everything, and enjoy watching both factions fake evidence for why their project is the biggest problem.)
On that theme: mix small and large goals. When faction A has failed to convince the party to attack a wizard's keep, they'll still feel satisfied that they recruited the nobleman's cellarer as a spy. A mix of goals lets everyone feel they can accomplish something every session.
Major-goal-failures don't end the game for either faction. This is important, or your party breaks apart. When the factions' goals for the party are opposed (or there's only time to accomplish one), lower the stakes - make sure the players don't think their goal is a life-or-death-necessity for their subgroup. Make sure the losing conspiracy can think "That hurt... oh well, better luck next time" and go on to the next scheme. You want the goal to seem to seem important... but not enough to kill the rest of the party over, thus losing their resources. (Real power groups rarely bet everything on one plan.)
It seems obvious to establish a note-passing rule for people to scheme secretly, but I advise against it. It brings the intrigue into an obvious, game-slowing mechanic. Instead, be sure to ask the factions secretly between sessions - not only what they plan to do next time (and what excuses they'll use), but also for what they just did. Give the players a limited power to retcon the explanations for their own actions, and they won't need notes, just cover stories.
("When I took that bandit prisoner off to the city guard to be hung, I actually dropped him off with my family's militia for interrogation. Once we know where they hid the loot, our side can send soldiers to go get it.")
Take notes on those cover stories, they can lead to consequences later... making players scramble to make facts fit the story they'd already told the rest of the party.
Despite what's been said in some answers here, I think D&D can be used perfectly well. I've run this approach to good effect in several Star Wars and Werewolf games, which are just as conflict-oriented. The key is to ensure that neither faction's goals are well-served by the total collapse of the party.
Maintaining Cohesion
Even enemies aren't adversarial all the time. Be sure that both factions have some goals that the other group simply doesn't care about either way. You can create the feeling and tension of intrigue without risking a party feud when tasks fail. (Bob ends up running out of arrows to make a plausible excuse for visiting the markets before storming the overlord's tower; actually he just needs a chance to slip his side a magic statue they gave him to transport.)
Let the factions disagree on methods or outcomes, not primary objectives.
Example: I ran a two-party competitive Star Wars game in which the players were Rebels and Smugglers. Both want to smuggle firearms past the Empire... but subgoals vary. When the stormtroopers pour out, they can fight together as a cohesive party. But if the stormtroopers attack some other random facility... the rebels want to act, and the smugglers want to lay low and get paid.
Example: Factions A and B wanting different candidates to inherit the throne is bad; one's gain is another's loss. Faction A wanting more money for the military and B wanting a balanced budget is better. There will be conflicts, but also some situations will make both factions happy (more farm tax income), and some will make neither happy (loss of expensive equipment). They may scheme to get different people on the throne... but that won't be the point for them, just a means, so the party doesn't have to break apart over it.
Example: Let a mage in faction A have research notes he's not using, that faction B would love to have. Make sure faction B's thief hears information that can prevent (or cause) blackmail attempts against A's priest. Let A have all the rich characters, and B always need money, but have much better contacts.
Major villains should be existential threats to both factions. (This needn't spoil the theme of intrigue; a new player trying to turn one faction into attacking the other is an enemy of both... and both sides may have independent reasons to deal with it.)
Be cautious with your intention to put the experienced players in one faction and the new ones in another; it leaves you without a character present to 'smooth the way' if the newer players struggle with a goal. Consider putting one experienced player in each faction, and letting them 'recruit' the rest of their group in-game.