It is a "display of confidence", meaning "submit myself to something unpleasant if I prove to be wrong" as in:
If the train arrives on time, I'll eat my hat.
To say "afterwards I had to eat my hat" is to admit that you were wrong.
http://www.phrases.org.uk/meanings/eat-my-hat.html
It's a metaphor. The metaphor has long since passed into regular speech, so most people aren't conscious of making it, but it's there nonetheless.
The face value of a bank note (or coin, postage stamp, etc.) is the value printed on the item. A five-dollar bill claims to be worth five dollars, so that's its face value. In other words, the face value of something is its apparent worth; its real value may not be the same as its face value.
Applying this term to words, if you take someone's words at face value, you're not questioning their apparent worth. You're assuming that the words are worth what they appear to be. As a result, you don't question them or examine them closely to find any flaws, inaccuracies, or untruthful statements.
If you don't take their words at face value, the opposite is true. You haven't assumed their words are correct or truthful. You might question them or examine them closely to find flaws, inaccuracies, or untruthful statements.
Sometimes this phrase is used to imply someone might be wrong; in other cases, this phrase is used to imply deceit or trickery. Which implication is appropriate probably depends on context.
Let's look at your context:
Don't take my words at face value.
If I tell you not to take my words at face value, I'm probably telling you I might be wrong, not that I might be tricking you. Why? Because if I were trying to trick you, I probably wouldn't give you any warning!
Best Answer
If there is an uncomfortable fact that you knew was true but you tried to avoid it, but can't avoid it anymore, you'd say
For example, you and someone else were driving a car, and then the car broke down in the middle of nowhere. You're trying to get the car to start, but it obviously isn't working. After a while, you give up and say:
It comes from "facing the facts". There are facts that you know are true, but you're trying to ignore them, so you've "turned your back against the facts". When you admit the facts, you're "turning around to face the facts". "Let's face it" means "Let's admit that these facts are true(even though we both knew it already)".