Yes, but with a strong emphasis. Perhaps it could be better thought of as "you are lacking some information, and if you had it you would no longer want what you currently want".
It certainly can be rude.
It can be less rude, depending on how reasonable it is to infer what someone really wants. E.g. if someone says "I want to do X so that I can accomplish Y" and you know that X will not actually help Y, then it's reasonable:
I want to clear my phone's applications' cache automatically, so it'll have more memory available and run faster.
You don't want that, because if the applications can't use cached data, they'll all run slower.
Now, I'd say this was a bit brusque, but not terribly rude, but this is different to someone having a problem with an application that has a bug with what it chooses to cache, and hence really do want to clear it. We know they don't want to clear the cache because it would hinder their actual goal that they want even more. (Or to think of it another way, we're not telling them what they want, they've told us what they want, and we're pointing out an implication of that).
It can be less rude when we know of something they might not reasonably expect.
Let's eat in that place across the road.
You don't want to eat there, I've seen the kitchen.
OR:
You don't want to eat there, everything's really spicy. (Because I happen to know that this person hates spicy food).
This last example raises a point that the degree to which we can reasonably say "you don't want" or indeed, "you want" relates to how well we know them; sometimes we can make such a judgement based on what information is only present in the communication at hand, sometimes because we have further background information.
And by extension, claiming someone doesn't want something can imply that we know them better than we do, and hence be over-familiar and therefore rude.
In all, it can be not rude at all, rude because of being over-familiar, brusque (not downright rude, but not particularly polite either) or downright rude, depending on context.
The comedian Harry Enfield used to have a character alternatively named for his catchphrase "Only Me!", his other catchphrase "You don't want to do it like that!" or his regular punchline "Now, I don't believe you wanted to do that!" the humour of which was entirely derived from the fact that much of the time, telling somebody they don't want to do something is indeed rude and/or annoying. Sadly the only two clips I can find of him are variants on the main premise where he actually gets on well either finding agreement or else helping in several cases where his advice was indeed reasonable (e.g. it really is true of most people that they "don't want to test the trigger of their new loaded shotgun while simultaneously licking out a small speck of dust from the barrel with their tongue"). In these clips the humour is because the audience are used to the character being annoying, rude and often actively harmful, and so they aren't good examples. IIRC, the character was killed off in a death scene in which he was informing a hangman that he didn't want to use the type of knot he had tied.
"I'm opt for the party" is ungrammatical.
"up for" and "down for" with respect to parties and activities in general mean the same thing: I want to go do it.
The phrase "To be up for something" means
to feel that you are strong or healthy enough to do something
The phrase "To be down for something" is more of a slang term and I always picture it as like when you've put your name down on the list indicating you want to participate then you are down for it.
Note that being "down with" something is a little different and means more like you are in agreement with the sentiment or idea being expressed.
Best Answer
Neither of them 'works for me', but they do represent the different kinds of semantic changes that English words often undergo.
Starting with 'work for': The verb 'work' has a huge number of meanings. One of these is an intransitive use: "It works! It works!" This means that some action or method has had a successful result.
Phrases like 'It works for me' come from this sense, but have an extra meaning of 'It produces the result I desire'. This is often used to suggest cures for illnesses or problems such as hiccupping: "Stand on your head and drink a glass of water, that always works for me."
In the sentence you give, however, as user162097 says, it now seems to mean something like 'it suits you'; i.e. it is a good match for your appearance or your personality or whatever. This is a new use of 'work for'. It's sort of a semantic extension to 'the result that I want'
In fact, it seems that nowadays just 'work' can have this meaning. In the movie 'Out of Sight' for example, there is a scene where a character who is robbing someone's house picks up the victim's expensive clothes and starts holding them up in the mirror to see if they 'suit him', muttering 'oh yeah, that's working, that's working." To me, this is just weird; maybe I'm old, or maybe I've been out of the States too long.
The second example is not really a new usage; it's just extending the range of the original meaning. X 'goes with' Y means that x and y are a good match, i.e. they are compatible. This is often used with fashions or colors: "Are you crazy? Red pants don't go with yellow socks." I think that 'The nickname goes with you' is using this meaning, as if a nickname were something you might wear. You could say this is a metaphorical use, but I would never say this either. Red pants and yellow socks, yuck.