Beauty is in the eye of the beholder
I’ve always found that the most difficult problem with appearance stats is that it’s supposed to be one single stat, that applies to all people. Even within the human race, there are people attracted to others that I’d personally find ugly, even repulsive, and people who would find those I am attracted to just as off-putting. That’s not even getting into sexuality, and then on top of that you have the issue of very different races with potentially massively different standards of beauty.
And then different appearances can be positive or negative depending on context. A drop-dead gorgeous person (for your own definition of drop-dead gorgeous) can be intimidating, scary, threatening. Particularly if that person seems to be using their appearance to their advantage. When you want someone to give you good advice, you want someone who is older, who appears to have been through a lot. When you just want comfort, someone older, but warmer and friendlier, like a kindly grandparent, is what you want to see. And these qualities are not what most people are looking for in those they’d take a sexual interest in.
We expect doctors, lawyers, and businesspeople to look certain ways. We expect models and such to look very different ways. One of the former looking like the latter makes us take them (unfairly) less seriously; one of the latter looking like the former (even if they are still attractive) often damages their allure.
And all of this is cultural as well as personal.
So it’s really, really hard to quantify all that. You cannot apply a single number to be someone’s appearance. You can’t even apply a single number to a given sex, or one for those who are sexually attracted to your sex and another for those who are not. Because sex isn’t always what appearance is about. You need a different number for every different context and for every different observer.
You cannot run all those different numbers; it wouldn’t just dominate a character sheet, or require a separate sheet to itself, it would require several separate sheets, one for each race, perhaps. And would probably still be lacking.
You can, obviously, choose to abstract away a lot of this. After all, d20 combat is massively abstract, and that’s a lot of the premise of the game. But perhaps because I have no combat skills or experience, but lots of social experience, I have never found such abstractions satisfying.
A series of conditional bonuses and penalties
Not written down, not specifically recorded, but literally what I have described: the d20 system encourages the use of “Circumstance” bonuses or penalties, to be adjudicated by the DM on the fly. Use these to determine how much someone’s appearance matches what the target wants from them. If you’re seducing them, you get a bonus if your body is one they find sexually appealing. But if you’re trying to give them advice, that’s not the issue at all; you instead get a bonus for appearing knowledgeable, sincere, and honest.
And these bonuses can apply to Bluff, Diplomacy, Disguise, and Intimidate. Being gorgeous might distract one person, and give you a bonus to Bluff, while it makes another suspicious and actually gives you a penalty. The key, as DM, is to be fair and consistent – give the appropriate bonus or penalties for a particular NPC.
The books recommend values from −2 to +2. That may not be sufficient to describe what you want. But I would caution against making appearance dominate the conversation: even when it is fitting, how you use it tends to matter more than the simple fact that you do appear a given way.
Expanded skill usages
Knowledge (local) can give you an idea of what a particular culture wants to see. It doesn’t always help – plenty of people eschew “standards” of beauty – but it can help.
Sense motive can give you a more personal sense of how a person reacts to different examples of beauty.
Low-DC (10-12) disguise checks can be used to match that standard. Higher-DC checks can progressively maximize that effect, or allow you to match more exotic tastes. Humans, we know, can use disguise to make themselves appear to be elves, but perhaps they can also use disguise to take on elven fashions and tastefully and skillfully make them their own, to make themselves look appealing – possibly even more so than actual elves, by dint of being exotic – to elves. Or to match, but make your own, the orcish markings and trophies of chieftains or shamans: not trying to fool them into thinking you are an orc, but to demonstrate gravitas and importance on a level they may not even consciously notice, but will give perhaps undue weight to.
Stock, mutually-exclusive descriptions
These can be freely chosen, to give players options and keep you clear on what they are going for. Basically, everyone can freely choose to describe themselves according to these terms, and you will use that to drive the conditional bonuses. Helpful for making things a little more “mechanical,” which many players find comforting and interpret as that aspect being taken seriously.
For examples, your appearance might be one of, say, sexy, elegant, ugly, frightening, serious, or silly, in particular by the standards of some race. Usually your own, but perhaps you are specifically gearing yourself towards the standards of some other race (maybe one where you were raised, or have trained to be an ambassador to, or to infiltrate and spy on). There are no numbers associated with this, just stock terms that can be used in a description, with examples of cases where that works for you and when it works against you.
Or you could let players choose multiple descriptors, but the more they have the more muted the effect is. Someone who puts just sexy as a descriptor would be able to seriously derail and distract those who are attracted, but would have a hard time “turning the sexy off” when it was inappropriate, and may find that some assume they are nothing more than a pretty face and take them less seriously. Someone who puts both elegant and serious would be able to turn on the charm and run a bit of seduction quite well, while also being able to turn around and lead a negotiation, but won’t have the sheer head-turning, mouth-dropping impact of the one who chose just sexy.
Best Answer
TL;DR: Charisma is often connected to appearance, but has no inherent association with any physical qualities--and definitely does not imply comeliness. The repeated references to Charisma as a physical attractiveness score are incomplete shorthand, referencing the most common use of Charisma for player characters.
Since you're focusing on 3.5-like systems, I'll quote from the best official 3.5 source of Charisma description I've ever found: the 3.5 Monster Manual entry "Ability Scores."
But first, I'd like to point out that this entry describes Str, Dex, and Con as "physical abilities" and the other three as "the remaining abilities" (in other places, Int/Wis/Cha are called "mental ability scores").
From this, it is clear that Charisma is not a beauty stat: an exceptionally ugly character might have that appearance represented by a high Charisma too. However, if we read further into this paragraph we find that Charisma is not technically a measure of physical appearance at all.
Strictly speaking, Charisma represents a creature's interpersonal skills. That is, a creature with high Charisma knows how to use its beautiful or hideous appearance to influence others. A terribly ugly or astonishingly pretty creature that doesn't have control over how its features influence others has low Charisma, and a man with an ordinary face who is masterful at influencing people has high Charisma. Remember, Charisma is the base stat for both Intimidation and Diplomacy.
ASIDE: The D&D Fourth Edition description of Charisma says the same thing in many fewer words.