I think you hooked up neutral correctly. Stay away from the 3-way switch.
Your assumption that cable 2 comes from the panel may be incorrect. Mechanical switches are direction agnostic, they don't care which is supply-hot and which is switched-hot. Smart switches do. The switch needs supply hot on its (presumably) black. Then, it will power up even without hooking up the red wire.
So I would test the black wire to either one of the blacks coming in cable 1 or 2. When the smart switch powers up and becomes recognizable, declare victory and hook the red wire to the other black.
This is not as hard as it looks
We can tell from your photo that the far left cable provides the always-hot and neutral for the lighting branches on the left and center switches, which leave via the two middle cables, while the far right cable is for the 3-way circuit and thus needs to be left alone. Furthermore, we can see the neutral bundle for the two outdoor light circuits in the back of the box, running left to right.
This means that putting both outdoor lights on the same smart-switch is easy. The neutral bundle gets pulled out from the back of the box and the switch's neutral gets added to it, with a pigtail of appropriate gauge wire used to make the connection. Likewise, the existing ground bundle is pigtailed to the ground screw on the switch, and another pigtail is run from that ground bundle to the ground screw on the 3-way switch to the right, as the grounding in this box appears to be improper.
Then, we can cut the wire that connected to both the left and center switches (on their lower screws in your photo) back to just before the first stripped section, strip an end, and connect it to the LINE terminal on the switch. The two wires going off to the outdoor lights are then wirenutted to a pigtail that connects to the LOAD terminal on the switch. Get a suitable faceplate (3-gang with a blank in one position, a decorator or toggle in the second, and a toggle opening in the third -- modular faceplates are also an option for this, or you could use a switch-filler in the existing faceplate if your new smart switch uses a toggle style handle), button it all back up, turn the breaker on, and enjoy your new smart switch!
Best Answer
I think you have your black and brown reversed. The ground green wire is connected to a black and to the box so other than color it is correct. The copper hanging out from the wire nut at the black brown, 2 blacks and a brown is that making a good connection? If it is I would put the black on the always hot and the brown on the hot to the light, I looked online for a schematic and only found black to black and brown to brown. That did not help. Since your original switch only had 2 terminals on , off control if wired the same way reverse them so the switch is being powered and it should work.