So, after some further fooling, I rewired it again as I did the first time (with a pigtail going from the neutrals to the silver terminal), the hot line in going to the top right black terminal, and the load going to the bottom right brass terminal. I turned on the breaker and again, the pilot light is on 100% of the time, whether the switch is in the on or off position.
So, I then swapped the hot line in, and the hot load out, and turned back on the breaker, and it now works properly. So, it looks like the directions are SORT OF correct, but Leviton put the line in where the load out should be, and vice versa. Basically what they say should be connected to the black screw should be on brass, and brass should be on black (still a DIYer, I never know the proper names for these things). I hope this helps someone else who buys this same switch.
Because you are replacing the switch, not installing a new switch, you can get away without a ground. You should install a nonconducting, noncombustible faceplate though.
NEC 2008
404.9 (B) Exception. Where no means exists within the snapswitch enclosure for connecting to the equipment grounding
conductor or where the wiring method does not include or provide an
equipment grounding conductor, a snap switch without a connection to
an equipment grounding conductor shall be permitted for replacement
purposes only. A snap switch wired under the provisions of this
exception and located within reach of earth, grade, conducting floors,
or other conducting surfaces shall be provided with a faceplate of
nonconducting, noncombustible material or shall be protected by a
ground-fault circuit interrupter.
If you were installing a new switch, you would be required to provide an equipment grounding conductor at the outlet. And the switch would have to be properly grounded, in accordance with the National Electrical Code (NEC).
There was a time when an equipment grounding conductor was not required at each outlet, so it's fairly common to come across this situation (especially when working in older homes). You'll often see exceptions like this written into codes, so as not to require a full rewire just to replace a switch.
New info, Better answer
If you are replacing a switch a ground is not required, as per the above exception. However, if you're installing a switch; replacement or otherwise, into a metal box that is grounded. The switch will be ground via the devices yoke and mounting screws. So if the metal box is grounded, the switch is also grounded.
If the box is nonmetallic, and there are other grounded devices within the same enclosure. You can ground the new switch using a jumper between the switches grounding screw, and the other devices grounding screw. Just keep in mind, that you can't terminate two conductors under a single screw terminal. So if you do this, you'll have to use pigtails to make the connection between the devices.
Best Answer
Yeah, it's real common inside junction boxes to have a lot of other stuff going on besides just your switch. Sometimes the wires that go to your switch also need to go elsewhere.
These multiple wires have to be spliced together somehow. A lazy/easy way to connect 2 wires and also a switch, is to use the switch as a splice block. In your case by attaching a wire to the screw and another to the backstab. Most of the current probably goes through the switch and out the other wire.
Like Ed Beal discusses, backstabs aren't very good. It may be better to go to a more traditional splicing technique, like a pigtail and wire nut. That will also make the wiring appear more logical.