I got sick of waiting for an answer to my broader question regarding how the damage multiplys in general, so I did a bunch of testing myself.

Method:

All tests were done by shooting a friendly setler in the back of the torso at close range.

The NPC was wearing nothing but a Minuteman outfit and had their HP set to 5000 with the console. I did not check what their Damage resistance was nor did I modify it, though I probably should have.

A single shot was taken, NPCs HP remaining checked with the console, then the game was reloaded. This was done 5 times.

Due to perks etc, my Sneak attack bonus was at 4.2 Not sure what my exact critical bonus is, I had 4 ranks of Tesla Science (+20% crit damage) and no other perks or mods which would increase it.

Tests performed with a Laser Rifle, instigating tests performed with Old Faithful with the same mods. Both showed a base damage of 144 in the Pip-Boy interface.

---

Results:

Normal Shot

111.81, 214.99, 213.32, 214.57, 212.48

Average: 204.72 damage (Average may be low? First shot did much less damage)

Critical Hit:

350.93, 349.86, 350.63, 350.34, 351.00

Average: 350.552

Sneak Attack:

905.80, 905.97, 906.70, 906.38, 905.46

Average: 906.062

Sneak Attack + Critical Hit:

1040.25, 1042.20, 1040.83, 1038.45, 1040.27

Average: 1040.40

Normal Shot + Instigating Weapon

425.43, 430.58, 431.57, 430.70, 352.29

Average: 414.114 damage (Average may be low? Last shot did much less damage)

Critical Hit + Instigating Weapon:

565.44, 559.52, 566.39, 565.26, 566.12

Average: 564.546

Sneak Attack + Instigating Weapon:

1810.25, 1812.87, 1813.63, 1813.38, 1812.00

Average: 1812.426

Sneak Attack + Critical Hit + Instigating Weapon:

1944.69, 1946.42, 1947.35, 1947.74, 1947.11

Average: 1946.662

---

Discussion:

The above results show some variance, with some shots noted as being much lower than others. It is not clear if this was due to poor accuracy causing some bullets to hit the wrong location (though injuries on target suggested all shots hit the back), natural RNG spread of damage, or damage resistance of the target.

When roughly comparing Sneak attacks performed with both Instigating and normal weapons, the stated Sneak Attack bonus does appear to be multiplicative with Instigating ( [Weapon Damage x2] x4.2) or at least close enough to satisfy me that this is the case.

Interestingly, Critical Hits do not appear to be multiplicative with Sneak and Instigating. In all situations the additional damage inflicted was close to the weapons stated base damage (144, give and take some variance). This suggests to me that critical strikes are seperate damage which is calculated and then added after sneak attacks and instigating are multiplied ( {[Weapon Damage x2] x4.2} + [Weapon Damage xCrit Modifier])

The Wiki suggests that Critical Hits use the following formula:

Ranged: Critical Hit Damage = DamagePaper + DamageBase x CriticalMultiplier

Melee/Unarmed: Critical Hit Damage = DamagePaper x 1.5 + DamageBase x CriticalMultiplier

DamageBase = Weapon damage, including workshop mods, but excluding perks and other bonuses. DamagePaper = Damage displayed in PipBoy (PipBoy cuts off decimal places)

Conclusion:

Sneak Attack and Instigating are multiplicative with each other.

Critical Hits are not multiplicative with Sneak or Instigating

This suggests: ([Weapon Damage x Instigating] x Sneak Modifier) + (Weapon Damage x Crit Modifier)

It's a little complicated.

I ran over 50 tests (20 modded weapons of 6 different types from 2-3 firing distances) before running out of time and patience. If you test other weapons or find this is simpler than it seems, please let me know.

Summary

Based on testing so far, VATS considers weapon fire to be:

• neither sighted nor hip fire for certain weapon types
• sighted fire for certain weapon types
• hip fire for no weapon types

The following have no effect on whether VATS treats something as hip fire or sighted fire:

• whether you're looking down the sight just before hitting the VATS key
• whether or not the VATS animation shows you looking down the sights

And finally:

• sighted fire may be treated very differently from scoped fire

So two very general rules if you use VATS often:

1. Prefer mods that improve sighted fire rather than hip fire.
2. In some cases the sighted version of a weapon (with a sighted mod) will perform significantly better than the scoped version (with a scoped mod).

Details Behind the Summary

Method

I de-modded weapons of 6 types down to their base configurations.

Then I remodded them as follows for hip fire, sighted fire, and/or scoped fire, adding only mods that directly mention affecting hip/sighted/scoped fire. Note: If a weapon used Scoped mods, I attached a scope; if it used Sighted mods, I attached a reflex sight.

Weapon configurations tested:

• 10mm Pistol: plain, hip, sighted, scoped
• Combat Rifle: plain, hip, sighted, scoped
• Hunting Rifle: plain, hip, sighted, scoped
• Assault Rifle: plain, sighted, scoped (no mods improve hip fire)
• Institute Pistol: plain, sighted, scoped (no mods improve hip fire)
• Double-Barreled Shotgun: plain, sighted (no mods improve hip fire; no scope)

I VATS-targeted a stationary settler from two predetermined distances and checked the to-hit percentage for the target's core/body. Disclaimer: no settlers were harmed in the making of this answer.

Key to Results

• "VATS Sighted" means the sighted variant had a higher to-hit chance than plain
• "VATS Scoped" means the scoped variant had a higher to-hit chance than plain
• "VATS Neutral" means plain, hip fire, sighted, and/or scoped all had the same to-hit within 2%

Results

• VATS Neutral: 10mm Pistol, Double-Barrel Shotgun
• VATS Sighted and very slightly VATS Scoped: Combat Rifle, Hunting Rifle, Institute Pistol
• VATS Sighted and very significantly VATS Scoped: Assault Rifle

Finding: Sighted ≠ Scoped

Initially I assumed that sights and scopes were treated the same. But it appears that sighted and scoped mods don't benefit sights and scopes uniformly. For example:

• A scoped hunting rifle with scoped-fire-enhancing mods performs only slightly better than an unmodded hunting rifle (e.g., 55% vs. 61%).
• A sighted hunting rifle with sighted-fire-enhancing mods performs much better (e.g., 55% vs. 85%).