I got sick of waiting for an answer to my broader question regarding how the damage multiplys in general, so I did a bunch of testing myself.
Method:
All tests were done by shooting a friendly setler in the back of the torso at close range.
The NPC was wearing nothing but a Minuteman outfit and had their HP set to 5000 with the console. I did not check what their Damage resistance was nor did I modify it, though I probably should have.
A single shot was taken, NPCs HP remaining checked with the console, then the game was reloaded. This was done 5 times.
Due to perks etc, my Sneak attack bonus was at 4.2
Not sure what my exact critical bonus is, I had 4 ranks of Tesla Science (+20% crit damage) and no other perks or mods which would increase it.
Tests performed with a Laser Rifle, instigating tests performed with Old Faithful with the same mods. Both showed a base damage of 144 in the Pip-Boy interface.
Results:
Normal Shot
111.81, 214.99, 213.32, 214.57, 212.48
Average: 204.72 damage (Average may be low? First shot did much less damage)
Critical Hit:
350.93, 349.86, 350.63, 350.34, 351.00
Average: 350.552
Sneak Attack:
905.80, 905.97, 906.70, 906.38, 905.46
Average: 906.062
Sneak Attack + Critical Hit:
1040.25, 1042.20, 1040.83, 1038.45, 1040.27
Average: 1040.40
Normal Shot + Instigating Weapon
425.43, 430.58, 431.57, 430.70, 352.29
Average: 414.114 damage (Average may be low? Last shot did much less damage)
Critical Hit + Instigating Weapon:
565.44, 559.52, 566.39, 565.26, 566.12
Average: 564.546
Sneak Attack + Instigating Weapon:
1810.25, 1812.87, 1813.63, 1813.38, 1812.00
Average: 1812.426
Sneak Attack + Critical Hit + Instigating Weapon:
1944.69, 1946.42, 1947.35, 1947.74, 1947.11
Average: 1946.662
Discussion:
The above results show some variance, with some shots noted as being much lower than others. It is not clear if this was due to poor accuracy causing some bullets to hit the wrong location (though injuries on target suggested all shots hit the back), natural RNG spread of damage, or damage resistance of the target.
When roughly comparing Sneak attacks performed with both Instigating and normal weapons, the stated Sneak Attack bonus does appear to be multiplicative with Instigating ( [Weapon Damage x2] x4.2) or at least close enough to satisfy me that this is the case.
Interestingly, Critical Hits do not appear to be multiplicative with Sneak and Instigating. In all situations the additional damage inflicted was close to the weapons stated base damage (144, give and take some variance). This suggests to me that critical strikes are seperate damage which is calculated and then added after sneak attacks and instigating are multiplied ( {[Weapon Damage x2] x4.2} + [Weapon Damage xCrit Modifier])
The Wiki suggests that Critical Hits use the following formula:
Ranged: Critical Hit Damage = DamagePaper + DamageBase x CriticalMultiplier
Melee/Unarmed: Critical Hit Damage = DamagePaper x 1.5 + DamageBase x CriticalMultiplier
DamageBase = Weapon damage, including workshop mods, but excluding perks and other bonuses.
DamagePaper = Damage displayed in PipBoy (PipBoy cuts off decimal places)
Conclusion:
Sneak Attack and Instigating are multiplicative with each other.
Critical Hits are not multiplicative with Sneak or Instigating
This suggests:
([Weapon Damage x Instigating] x Sneak Modifier) + (Weapon Damage x Crit Modifier)
Best Answer
No, it doesn't appear to do so. I tested both unarmed and with a pool cue against a group of six ghouls putting one hit on each and did the same damage to each of them.
Other players have tested this, too and can verify that the Gun-Fu perk doesn't affect melee weapons.