It's a little complicated.

I ran over 50 tests (20 modded weapons of 6 different types from 2-3 firing distances) before running out of time and patience. If you test other weapons or find this is simpler than it seems, please let me know.

Summary

Based on testing so far, VATS considers weapon fire to be:

• neither sighted nor hip fire for certain weapon types
• sighted fire for certain weapon types
• hip fire for no weapon types

The following have no effect on whether VATS treats something as hip fire or sighted fire:

• whether you're looking down the sight just before hitting the VATS key
• whether or not the VATS animation shows you looking down the sights

And finally:

• sighted fire may be treated very differently from scoped fire

So two very general rules if you use VATS often:

1. Prefer mods that improve sighted fire rather than hip fire.
2. In some cases the sighted version of a weapon (with a sighted mod) will perform significantly better than the scoped version (with a scoped mod).

Details Behind the Summary

Method

I de-modded weapons of 6 types down to their base configurations.

Then I remodded them as follows for hip fire, sighted fire, and/or scoped fire, adding only mods that directly mention affecting hip/sighted/scoped fire. Note: If a weapon used Scoped mods, I attached a scope; if it used Sighted mods, I attached a reflex sight.

Weapon configurations tested:

• 10mm Pistol: plain, hip, sighted, scoped
• Combat Rifle: plain, hip, sighted, scoped
• Hunting Rifle: plain, hip, sighted, scoped
• Assault Rifle: plain, sighted, scoped (no mods improve hip fire)
• Institute Pistol: plain, sighted, scoped (no mods improve hip fire)
• Double-Barreled Shotgun: plain, sighted (no mods improve hip fire; no scope)

I VATS-targeted a stationary settler from two predetermined distances and checked the to-hit percentage for the target's core/body. Disclaimer: no settlers were harmed in the making of this answer.

Key to Results

• "VATS Sighted" means the sighted variant had a higher to-hit chance than plain
• "VATS Scoped" means the scoped variant had a higher to-hit chance than plain
• "VATS Neutral" means plain, hip fire, sighted, and/or scoped all had the same to-hit within 2%

Results

• VATS Neutral: 10mm Pistol, Double-Barrel Shotgun
• VATS Sighted and very slightly VATS Scoped: Combat Rifle, Hunting Rifle, Institute Pistol
• VATS Sighted and very significantly VATS Scoped: Assault Rifle

Finding: Sighted ≠ Scoped

Initially I assumed that sights and scopes were treated the same. But it appears that sighted and scoped mods don't benefit sights and scopes uniformly. For example:

• A scoped hunting rifle with scoped-fire-enhancing mods performs only slightly better than an unmodded hunting rifle (e.g., 55% vs. 61%).
• A sighted hunting rifle with sighted-fire-enhancing mods performs much better (e.g., 55% vs. 85%).

I found the ammo ID: it's 65df3, and the weapon ID is 65df2.

If you open the console command and type "player.additem 00065df3 [amount]", you will have ammo for the nuke weapon.