First increase the damage, then apply reductions

The developers have explained how those abilities should work, based creatures that appear on certain adventure paths and monsters that do have that combination of abilities on the bestiaries.

In Mummy Mask #1 (The Half-dead City), there is a clay creature that has a vulnerability to fire, and being a construct, they have hardness , but first you increase the damage by 50%, then you apply hardness, then anything that goes through is normal damage, as explained by the author of the adventure and later confirmed by Robert G. McCreary (a developer).

The animated objects are constructs, so they are now creatures, not objects. As creatures, energy damage is not halved against them (in effect, becoming a creature trumps the normal object rules). So they take full damage from energy attacks (150% if they are vulnerable to that energy type), then hardness is applied. However, page 174 of the Core Rulebook states (under "Vulnerability to Certain Attacks") that "Certain attacks are especially successful against some objects.In such cases, attacks deal double their normal damage and my ignore the object's hardness."

Even though the animated objects are no longer "objects," I would still apply this rule about overcoming hardness to them in this adventure (but the 150% creature vulnerability trumps the double damage to an object part of the rule), as hardness can be quite difficult to overcome for low-level PCs. So if the PCs use fire against the warrior dolls, for example, it would deal 150% the normal damage, and ignore the hardness. Other energy damage would deal full damage which would ten be reduced by hardness, the same as for any other attack against it.

Another example are the robot creatures from the Iron Gods adventure path, where they have linked to McCreary's answer several times when questioning the order of abilities and again confirmed by James Jacobs on his unofficial thread, because all robots are vulnerable to electricity, but some of them are also resistant to electricity, like the iconic Annihilator from the Inner Sea Bestiary cover, and even though they have hardness, they should not be treated as objects, just like other constructs.

You need to determine the total amount of damage the robot WOULD take before applying hardness and electricity resistance. So once the damage is determined, first thing you do is apply the vulnerability effect to get the total damage. Hardness and resistance kick in after that.

It is difficult to counteract homebrew rules/items

"Spell resistance" is only ever specified under the 7th level class feature of Oath of Ancients paladin granting resistance to all spell damage to anyone in their aura. This homebrew is basically combining that mechanic with magical resistance or the use of a Rare magic item like the mantle of spell resistance, but doing so outside of that particular class. Coming up with a counter for a homebrew mechanic that provides advantage on all saves AND resistance to all damage types from spells becomes difficult because the RAW mechanics don't really account for this.

There are mechanics like Heightened which force disadvantage on a save and the feat Elemental Adept that lets you ignore resistance for a specific damage element - but a total mechanic of resistance to all damage from spells or ignoring advantage on saves doesn't exist because that isn't a RAW ability in general.

You need to talk with your DM

This sounds more like an issue at the table than an issue with mechanics or using mechanics to counteract in-game difficulties. Talk to your DM about why they feel the need to do this. They're basically granting Magic Resistance to all monsters (and CR should be adjusted accordingly) as well as giving them each the potion of invulnerability, a rare item that only lasts a minute (but maybe extending the minute and giving them a 'free' action to drink it.)

Either way, there's a deeper issue that needs to be resolved. Working around it will unlikely be successful.