It sounds like the two black wire with the pigtail are the incoming hot and a branch hot to another location, such as the outlet. The other black attached to the switch is probably the switched hot that goes to the fixture being controlled.
You can verify this by turning the switch to off, making sure all the wires and terminals are clear and not touching anything else metal, and then turning on the breaker. Using the non-contact tester, carefully check the wires. The paired blacks should read hot, but the switched black should not.
If this is what you have, wiring the new switch is pretty straightforward. Turn the breaker off again. Confirm no wires are now hot. The new switch is basically connected the same way as the old, but with a neutral wire and ground wire added.
- The hot pigtail is connected to the line terminal.
- The switched black wire is connected to the load terminal.
- A pigtail (white) is added to the bundle of neutral wires and connected to the neutral terminal
- A pigtail (bare) is added to the bundle of ground wires and connected to the ground terminal.
The traveler terminal is not used (and it looks like it is covered anyway).
Use wire nuts, and if you like, tape over them for extra safety. You also could put a wrap of tape around the z-switch covering the terminals. Carefully insert the wires back in the box. Screw the switch into the box. Turn the power back on.
While most dumb switches do not need a neutral connection, many smart switches, such as the z-switch, do. Now all switches have a separate ground connection, although many in the past did not.
The black wire that goes to multiple switches should be your hot leg. If the black wire in the back is on the same side (top / bottom) as the wire going from the screw to the other lights this is the line and it will need to be connected to the other switches and your new one or they will no longer work if capped off if that is the case the other wire is the load. The whites are the neutrals when bundled (some times in a switch loop a single white can be hot, they are supposed to be marked but many times they are not).
Best Answer
There's more stuff on this circuit
The black wire you see running off to the top of the box is an always-hot that simply powers more things on the circuit, such as receptacles or other lights, while the red wire going up is the switched-hot to the light fixture this switch controls.
Cut and splice the white wire to the neutral from the smart-switch -- you'll have 3 white wires under this nut. Splice both black wires to the always-hot going into the smart switch. Splice the switched-hot from the smart switch to the red wire. Leave any 3-way or traveller terminals on the smart-switch capped-off or taped over.
As to the switch not fitting -- you'll have to finagle a single gang box extender (or a surface raceway starter box) into place and screw it to the box, then screw the switch to the extension. If the extender or starter box is nonmetallic, then you'll have to pigtail the ground on the switch to a ground screw on the back of the box, as well.