I am pretty sure that the rules never clarify beyond what you have already quoted, and I am certain that the rules you have quoted are not definitive on this point.

I, and literally everyone I have ever played with or talked to, have always assumed that casting spells with both material and somatic components takes one hand. Never really paid very much attention to it. But I suppose the assumption was either that the material component is consumed before the gesturing takes place, or else that the gesturing is done with the hand holding the material component.

The Grappled condition and the Grappling subsystem don't restrict the use of arms

It doesn't matter how many people you have grappling a target, the target will always be able to use their somatic components. If you want a system to model grappling as physically restraining an opponent's limbs, you'll need to develop one. 5e doesn't provide anything there. Since you can't grapple a limb, there's no situation where you'd need to figure out if it interferes with hand-holding, and no rules are provided.

Spells target whoever the caster wants them to target

The spell does require a caster to target a circle of willing creatures, but the caster, as usual, chooses what valid target to select for spellcasting. 5e does not have any system in place to make the players of spellcasters unable to choose the targets they'd like to select from the information at hand. A caster could choose a circle that included a grappling opponent, if they wanted to and the grappler was willing and the GM ruled they had linked hands properly, but they could also choose a circle without said grappler as long as it's a valid circle. The caster, not the spell, chooses the target.

Basically, grappling is not at all an effective way of shutting down spellcasting of any kind, least of all teleportation effects (including Plane Shift). Even if you take the Grappler feat so you can retrain an opponent and even if you also have Mage Slayer and have taken the three rounds worth of actions it takes to grapple a mage, shove them prone, and restrain them, they can still totally Plane Shift away no problem. You can't even hit them with your attack of opportunity, unless they decided to take you along for the ride (you get the attack after the spell happens).

Worst of all, with Plane Shift in particular, the spellcaster can, if they like, cast the spell on you instead, which will kill you if they succeed at a melee spell attack (that they make without advantage or disadvantage if they are restrained, since you are also restrained), since you will then find yourself in a random location on the Deadly Plane of Death and Very Slow Time with no way to return unless you also know Plane Shift.