It's not in the rules, but DMs have the latitude to allow it.
As you point out, I couldn't find anywhere in the rules to support an action like that. However, the system allows for DMs to make judgments about whether an action is reasonable/possible, as well as how difficult it is (PHB 192):
When you describe an action not detailed elsewhere in the rules, the DM tells you whether that action is possible and what kind of roll you need to make, if any, to determine success or failure.
This specific instance is probably balanced.
Thus, it's up to you as the DM to determine whether this specific case should be allowed. More specifically, we need to ask, "If creatures are allowed to do this, would they always do it?"
From a theorycrafting perspective, it seems roughly balanced. The effects of pinning a caster's arms (or any character's arms, really), is roughly comparable to Hold Person, which is a second-level spell. We can take a cue from Hold Person, and say that if we want the effect to last multiple rounds, the grabbed character should get an opportunity to escape every round (as if from a grapple). The rest of the balance is probably a wash, because such a pin prevents the attacker from doing anything else, but they can do it for free.
We can also look to the grappler feat, which does something similar (PHB 167):
You can use your action to try to pin a creature grappled by you. To do so, make another grapple check. If you succeed, you and the creature are both restrained until the grapple ends.
Consider that you're allowing any character to take the benefit of this feat for free, and that the effect you're trying to create is actually more powerful than this feat. I'm personally OK with this, given that my players don't really use feats, but it's ultimately up to you and your players.
I have used a similar rule in actual play, and it seems to be a bit underpowered compared to other possible actions. I had a few of my NPC characters try to grab and pin the PCs, both to prevent them from escaping and to incapacitate them. I required my NPCs to initiate a grapple with one action, and then make a contested strength check to pin them with another action. Because the process costs two actions, allows for two "saving throws", does no damage, and still permits verbal-only spellcasting, its opportunity cost is very high--imagine the damage that a strong creature could do in two rounds! Additionally, once they are pinned, the grappler cannot do anything else, which effectively takes them out of the combat. While such an action was appropriate for the situation, I cannot imagine a smart combatant wanting to do this with any frequency.
However, if you do include this rule, you should remember that it will affect your future combats. It might not make much sense to try this move if there are lots of enemies, but it can completely shut down a combat with a solo spellcaster, if they ever move within melee range. It will also open up this option to enemies restraining the PCs, which they might not be super happy about.
You must release the target to attack, probably ending the grapple
D&D 5e has a broad intent that any word which is not explicitly given special rules meaning, instead has its plain English meaning, though I don't have a citation prepared for it. In plain English we see this:
Using at least one free hand, you try to seize the target ... you can release the target whenever you like
While "release" technically has multiple meanings, I find this context unambiguous in having Release mean "stop holding with your hand". From this plain English reading, I think it's clear that between the time that you "seize the target" and the time that you "release the target" your hand is occupied by that target; in other words, in order to use both your hands for a 2-handed weapon attack or other purpose you must release the target.
There does remain some ambiguity as to whether that actually ends the condition though. The sentence in question states:
The condition specifies the things that end it, and you can release the target whenever you like
But there isn't a strict connection indicating that "release the target" is an additional thing, not part of the set specified by the condition, which ends the grapple. It does seem likely that it was intended to be so based on the juxtaposition of the concepts, and supported by the subsequent section which unambiguously states that the means of ending a grapple are not entirely restricted to those specified by the condition.
Regarding "Common sense"
In a comment you mentioned that you don't bring real-world common sense into D&D, which is fine. Sometimes the rules explicitly contradict reality; after all you can't complain about someone casting a fireball on account of "magic isn't real". However, D&D also isn't a computer Rules Enforcement System where ambiguous edge cases have a single answer that's always correct and non-negotiable. Instead, D&D is adjudicated by Some Guy who, in my experience, is much more likely to be swayed by "It doesn't make sense that my Half-Orc Barbarian is physically incapable of throwing a gnome" than by "the rules don't say that letting go stops the grapple"; your mileage may vary depending on the DM. In general, I would say that ambiguous situations not clearly covered by rules are more likely to take whichever interpretation most closely matches common sense even if other times common sense is ignored.
Best Answer
The Grappled condition and the Grappling subsystem don't restrict the use of arms
It doesn't matter how many people you have grappling a target, the target will always be able to use their somatic components. If you want a system to model grappling as physically restraining an opponent's limbs, you'll need to develop one. 5e doesn't provide anything there. Since you can't grapple a limb, there's no situation where you'd need to figure out if it interferes with hand-holding, and no rules are provided.
Spells target whoever the caster wants them to target
The spell does require a caster to target a circle of willing creatures, but the caster, as usual, chooses what valid target to select for spellcasting. 5e does not have any system in place to make the players of spellcasters unable to choose the targets they'd like to select from the information at hand. A caster could choose a circle that included a grappling opponent, if they wanted to and the grappler was willing and the GM ruled they had linked hands properly, but they could also choose a circle without said grappler as long as it's a valid circle. The caster, not the spell, chooses the target.
Basically, grappling is not at all an effective way of shutting down spellcasting of any kind, least of all teleportation effects (including Plane Shift). Even if you take the Grappler feat so you can retrain an opponent and even if you also have Mage Slayer and have taken the three rounds worth of actions it takes to grapple a mage, shove them prone, and restrain them, they can still totally Plane Shift away no problem. You can't even hit them with your attack of opportunity, unless they decided to take you along for the ride (you get the attack after the spell happens).
Worst of all, with Plane Shift in particular, the spellcaster can, if they like, cast the spell on you instead, which will kill you if they succeed at a melee spell attack (that they make without advantage or disadvantage if they are restrained, since you are also restrained), since you will then find yourself in a random location on the Deadly Plane of Death and Very Slow Time with no way to return unless you also know Plane Shift.