Single switches that control a light are called single pole.
When there are two switches that both can turn a light on or off, they are called three way. (I am not sure about the origin of this label, perhaps because it required three wires; it does not mean that there are three or more switches - those are called four way!) In those switches, flipping the on off control on either switch will change its state: if it was off, it will go on, and vice versa.
When a dimmer control is part of a three way setup, one switch usually controls both the level and on/off, while the other switch only controls on and off. Both switches need to be the three way type.
It sounds like someone has put a single pole switch in a system that needs two three ways. Part of the problem is that there needs to be the right wiring between the switches. There has to be two hot wires*. There also may need to be a neutral wire, depending on the wiring configuration.
To determine the solution, we need to know what the wires are in each box that holds the switches, and what wires are currently attached to the switches.
SUPPLEMENT
It is possible to place dimmers at multiple locations if you use certain digitally controlled dimmers. You still need wiring between the dimmers that are able to have two hot lines, called followers, as well as a ground wire. You also need to provide a neutral wire to the fixture. That sometimes is part of the cable that goes through the dimmer boxes, but not necessarily.
Best Answer
I figured it out after opening it up. The old switches the previous owner installed were actually upside down and the common terminal was in the wrong spot, so when I tired them up the same way the hot was only being properly fed by one switch and when that switch was off the other switch wouldnt work. Once I figured out that that was the problem it was an easy fix.