Sure, it can come across as rude, especially when it's uttered bitterly, and it's meant to imply, "Do whatever you want - I don't care."
However, in a more casual context, it's not necessarily rude.
Sales Clerk: "...and that's the main difference between the red widgets, and the blue widgets."
Customer: "Thanks for that information; I'd like to buy two of the red widgets."
Sales Clerk: "Suit yourself."
Even in that snippet, though, I do detect an undercurrent of "That's not the way I would do it, but, if you say so..." So, I suppose something more along the lines of:
Fine by me.
could be considered safer. Still, I woudn't consider "Suit yourself" to be anywhere near as inconsiderate as something more vulgar, such as "I don't give a rat's ass."
"Hell no" is a way of conveying strong emphasis in one's disagreement. As Berker mentioned, "politeness" is context dependent. You haven't mentioned any example usages in your question, so we are all left imagining where you might use this, trying to determine whether or not your utterance would cross the boundaries of politeness:
Let's saying I'm talking with a co-worker who I have worked quite closely with in the past.
How was your weekend?
It was okay. My daughter has had a boyfriend for a couple of weeks now, they wanted to know if they could go camping together on Saturday night.
With a group?
No, by themselves.
What did you tell her?
I told her, "Not only 'No,' but 'Hell, no.'"
That's an idiomatic way of saying there was absolutely no way I was going to let my daughter go camping with her boyfriend: not at her age, not at that stage of their relationship, not when I knew so little about her new boyfriend. I doubt my coworker would find the wording impolite, unless he was prone to take umbridge at expressions with even a modicum of religious overtones.
Now, let's move the conversation to another place in my office, where someone is asking if I've seen the monthly report yet:
Have you seen the monthly report yet?
Oh, is it out?
Yes, it came out yesterday. Did you see it?
Hell, no, I haven't seen it. I just told you, I didn't even know it had been released yet.
That remark would be considered rude – not because of the word hell, but because it's said in a much sharper tone than was necessary. My coworker was merely being polite, and I started acting like a jerk.
You use "Hell, no" (or "Hell, yes") when you want to convey strong emphasis. "Hell, yes, I'm glad Manchester won the game" works just fine when two football fans are talking about sports. "Hell, yes, I'm glad it's Friday" might be overdoing it when someone greets you at the office; I'd reserve that kind of talk for only my closest associates. Hell, yes, I'm glad to see you" is almost assuredly too much if I am meeting a business associate for the first time, but it might work just fine if I'm meeting my brother at the airport, and I haven't seen him for two or three years.
Best Answer
Note that what the hell (or the more common and even more offensive what the fuck) is a short form for something like...
So just omit the "potentially offensive" bit in the middle. According to context, it might be you need to say "What are you doing?", or "What is that?", but hopefully the general principle is clear.
Lots of people probably think of "hell" above as a "toned down" variant on "fuck". You could tone it down even more by saying "What the heck", or "What the blazes", for example, but increasingly I think such euphemisms are seen as somewhat dated. Either swear or don't - there's no point in pussyfooting around with expressions that are really just attempts to swear without literally doing so.
Arguably just a personal opinion, but I think as a rule non-native speakers may well be judged more harshly than native speakers when it comes to "offensive" language. I believe most people (not just parents) are more likely to be bothered by children swearing than by adults using the same language. Whether you like it or not, it's unavoidable that many people will (consciously or unconsciously) conflate "inarticulate" foreigners with native children who are also in the process of learning English.
Therefore, although I can understand why people learning English are often particularly interested in learning how to use swear words (hoping to sound more like native speakers), I think this is a misguided approach. Personally, I believe swearing effectively/"correctly" is quite an art - which even most native speakers never really master, because it's so much more context-sensitive than most language usage.