[RPG] What spell level should this homebrew spell have? (gain resistance to one damage type, but vulnerability to another)

For the purposes of the resistance/immunity example in the original question:

"bludgeoning, piercing, and slashing from nonmagical weapons that aren't silvered"

The "weapons" portion of the example is actually sufficient to know that it does not apply against the example sources of damage, magical or not. The resistance only applies to specific damage type subcategories of weapon damage. None of the given examples are weapon attacks, so this resistance/immunity to damage from weapons does not apply to the given non-weapon examples in the question regardless of what type of damage is being done.

That said, errata has updated most (all?) instances of the given resistance/immunity example in the question to the following:

Bludgeoning, Piercing, and Slashing from Nonmagical Attacks that aren't Silvered

(Weapons -> Attacks)

Now it's clearer that the magical status of the attack itself should be considered.

The Sage Advice compendium provides the following checklist for determining if something is considered magical (see also: How do I know if an ability is magical?)

If you cast antimagic field, don armor of invulnerability, or use another feature of the game that protects against magical or nonmagical effects, you might ask yourself, “Will this protect me against a dragon’s breath?”

[...]

Determining whether a game feature is magical is straightforward. Ask yourself these questions about the feature:

• Is it a magic item?
• Is it a spell? Or does it let you create the effects of a spell that’s mentioned in its description?
• Is it a spell attack?
• Is it fueled by the use of spell slots?
• Does its description say it’s magical?

Since the question's examples are all spells or spell attacks, they fall cleanly under the umbrella of being magical. As such, thorn whip and the initial spell attack of ice knife are considered magical attacks, which clearly bypasses the errataed resistance/immunity text. Damage dealt as an effect of spell that isn't related to a spell attack roll (like the damage of earth tremor) is both magical and not from an attack, so the resistance/immunity text also doesn't apply there.

It's also worth noting that there's currently no such thing as an attack that is both a spell attack and a weapon attack, as noted in answers to a question about how to refer to non-spell attacks.

I didn't do any math, just going off of experience building quite a lot of encounters here. If an exact calculation is what you want, leave a comment and I'll delete this answer.

Coinflip encounter

This encounter will come down to the initiatives rolled. With 13 AC and 28 hit points, neither of the swarms should live more than one attacking round from the party at level 6, even taking into account resistances. They should however take up approximately that round of attack by the players.

Which leaves the Living Spellbook, which gets a practical one round of free attacks because the players will be dealing with the swarms. If they're careless or unlucky (aka, you felt like giving them a hard time), the book will use two multiattacks on the sourcerer or artificer, who should go down fast. At that point the combat depends on whether they were able to dispel the mage armor. If the swarms got off an attack before dying, I'd say those chances are low. Otherwise they are high and the fight should be a breeze for the barbarian and fighter (the arcane explosion should do approximately zero agains them with a Strength saving throw).

So the 2 scenarios I see happening are:

Swarms have relative high initiative

• The swarms attack the artificer and sourcerer for about 50% of their HP
• The swarms die
• The Living Spellbook effectively takes out the artificer and sourcerer out of the fight
• Mage armor isn't dispelled, your barbarian and fighter have a hard time, but should barely win out before any deadly casualties. Bad dice rolls for the player result in a loss

Swarms have relative low initiative

• Swarms die immediately
• Sourcerer dispels mage armor on the Living Spellbook
• Living spellbook becomes a glass cannon with 14 AC, which both the barbarian and fighter easily break
• Fight is a breeze, biggest risk is a small paper cut by one of the twirling paper sheets