Phrases – Example of ‘I Couldn’t Fail Not to Disagree with You Less’

How about independent scholar?

"Call me anything you like, but don't call me late for dinner" is a fairly common (in the US, anyway) joke that's used as a social ice-breaker in moments of potential awkwardness:

• when one person is unsure which of several possible honorifics to use (imagine meeting, for instance, Madeleine Albright: do you call her Dr. Albright, Madam Secretary, Madam Ambassador, etc.?) The person being addressed can set the other person at ease by using this phrase to indicate that s/he is not overly concerned with titles.

• when two people have met at least once before, and one can't remember the other's name; the one who's been forgotten can use this phrase to soften the other's embarrassment.

The joke is (like most wordplay) a matter of establishing expectations and then dashing them, or (more to the point) a broken parallelism. In the first part, you're invited to call the person by first name/last name/title/nickname/whatever you like (but the expectation is that it will be some sort of name); in the second part, you're told not to call the person "late for dinner". Your initial reaction is that "late for dinner" is some sort of nickname, but in fact it means "call me for dinner - but on time, please!" The actual meaning is not important as long as it's unexpected, so there are a few variations; I've heard "just don't call me long-distance."

That's the context of the joke; I don't think that, in your case, I would infer any comment on your appearance. The person saying it probably just assumed you were familiar with the joke, and went straight to the punchline.