It sounds like you have power entering the junction box at the light fixture, then running to one of the switches. If you actually have power entering both switch boxes independently, this will never work and you need to look at the connections at the light fixture.
EDIT Here's a 3-way switch:
Here's what should be happening in your case in order for the switches to work:
One wire, entering one switch box, should be hot from the power source (if the circuit breaker is on, this wire is hot).
That hot wire should be connected to the common screw on the 3-way switch.
The travelers (the other two screws on the switch) should run straight from one switch to the other, with no other connections along the way, through the light fixture's j-box. (the travelers from each side will be connected to each other with wire nuts in the j-box, of course).
The wire that is attached to the common terminal on the other 3-way switch should be connected to the black wire on the light fixture. This is the power coming out of the switches.
The white wire on the light fixture should be connected (wire nut) directly to the neutral wire in the j-box under the light fixture. If you're using Romex, this is going to be the white wire in the 14-2 or 12-2 cable that supplies power into that j-box.
Here's another StackExchange answer that shows very simply how 3-way switches function.
And another very long, detailed answer (holy smoke, I thought my answer was too long).
This diagram shows 3-way switches with power entering the light fixture box, and seems less confusing than many. It's surprising how many mislabeled diagrams of 3-way switches there are out there:
http://cindyjerrell.com/artwork/2012554_3_Way_Switch_Wiring_Diagram.html
A little more explanation
There are different ways to physically wire up 3 way switches, depending on whether power enters one of the switch boxes or enters the switched fixture. But they all add up to exactly the same thing.
You have 3 terminals on each switch. The different-colored one (usually darker) is the common, and the other two are "travelers."
So you have 2 travelers. This is usually the red and white, if your wiring is NM/Romex 3-rope, and you would use a black permanent marker or wrap black tape around the end of the white wire to indicate it is a hot conductor (not a neutral) in the switch circuit.
It sounds like you have something other than NM 3-rope going on in the wall between the light fixture and one of the switches (not the end of the world--electricity doesn't care about the color of the insulation on the wire).
The travelers always run straight between the same-colored terminals on both 3-way switches, no other connections
Depending on the positions of the switches, one of the travelers is always hot.
If the switches are positioned so that the common on both ends is connected to the hot traveler (which could be either traveler), then the circuit is complete and the light comes on.
The purpose of the travelers is to complete or break an electrical connection between the common terminals on the 3-way switches.
Now that the travelers are out of the way (connect them to the two same-colored terminals on the switches, forget about them), consider the common terminals...
Regardless of where the power enters the circuit, you run your hot wire to the common terminal on one switch, and wire the hot lead on your fixture to the common terminal on the other switch.
The entire set of 3-way switches, as a whole, just switches the juice on the common terminal of the 3-way switch at the opposite end.
Managed to solve this myself with a live wire detector:
I believe the red/black on the far right of the diagram are wired incorrectly, as the black is live and the red neutral.
I believe this set up is described here, which states that it's not recommended for new installations for many reasons, but can be found in old homes.
Best Answer
First off, the wiring of the existing setup looks totally normal as depicted in the diagram.
It looks like you can use these new switches in the current wiring setup. But first it is necessary to understand the connections of the two right cables shown in the upstairs switch box. One of these goes off to the light fixture and the other is the incoming power cable.
Cut the power at the circuit breaker box and make 100% sure that the power is off with a non-contact circuit tester. Also good to verify with a volt meter as well.
Remove the twist on wire connector that joins the two black wires in the upstairs box. Also remove the black wire that connects to the current upstairs switch Common terminal. Make sure you position all the wires so that they are not going to touch and are safely away from any other terminals or the electrical box and grounds. Next step is to identify which of the two right most black wires comes from the power panel circuit breaker and which one goes to the light fixture. The safest way to do this is to use a wire tracer tool from the light fixture box. The alternative, but less safe, is to temporarily turn the power back on and use a volt meter to see which of the two black wires becomes energized with the AC mains voltage. Once determined turn the power back off and/or remove the wire tracer. Mark the correct cable as LIGHT and the other as POWER so as to not confuse which is which.
You will use the AUX switch in the downstairs location. Remove the old switch and connect the new one as follows: Red to Traveler terminal - White to Neutral Terminal - Safety Ground to switch ground and Black will be capped off and not used.
The RELAY switch will be used in the upstairs location. Remove the old switch and then follow the steps below to connect the new relay switch.
Note that I will refer to your two right most cables as the LIGHT cable and the POWER cable as you determined and marked in step 2. As drawn in your diagram the left cable is the one that routes to the downstairs switch box.
Connect the white wires of all three cables together along with a white pigtail. This will be a wire nut that has four wires in it. These will now connect the neutral of the incoming power cable to neutral for the upstairs switch, to the neutral return from the light fixture and neutral for the upstairs relay switch on the pigtail.
Connect the other end of the white pigtail to the Neutral terminal on the relay switch.
Connect the safety Ground from the relay switch into the three ground wire connections in the box from the three cables.
Connect the red wire to the Traveler terminal on the relay switch.
Connect the black wire from the marked POWER cable to the Line terminal on the relay switch. This should connect to nothing else.
Connect the black wire from the marked LIGHT cable to the Load terminal on the relay switch. This should connect to nothing else.
The black wire in the left most cable will not be connected to anything and should be capped off.
That should do it. You can double check your work, mount the switches to the box and restore power to the circuit.