Learn English – a course of — meaning

Both of the original poster's examples are correct and natural in Standard English. Which one is used depends on whether the person responding first thought to say "No", or first thought to say "Of course".

I can see why this would be confusing to you. At face value, those responses really have nothing to do with what you said to the person.

Idiomatically those answers have become common, routine responses to being thanked. They are just what people automatically say to respond to being thanked when they don't want to say, "You're welcome", because it's too formal.

However, there is an underlying assumption that can help you to understand the context and a meaning that you can infer. While they might not be aware of it, when people respond this way they are assuming you have a question in your mind when you say 'Thank you'. The assumed question they're responding to is something like

I'm sorry you had to go to the trouble of doing that for me. I wonder whether it was worth it to you to do that because now I'm in your debt and might not repay you. I hope you're not expecting something in return.

Whether or not you thanked them with that question in mind, that's the question that's being answered. With this knowledge you can understand the meaning of the response as something like one of the following

Yes, sure, it was worth it to me just to help you.
Of course I was happy to do it, or otherwise I wouldn't have.
Of course you don't owe me anything. I offered it out of kindness.

This also explains why people sometimes respond with

It was no trouble.
No problem.
Don't worry about it.
I'm happy to do it.

There might be many variations on this assumed question and the subtext of the answers that could explain what's going on, but the thing to understand is that there is unspoken meaning being assumed and expressed in the conversation.

Personally, I like to say, 'You're welcome'. To me it's more gracious and I think people appreciate the formality of it. It assumes less about what the person thanking me is thinking. The underlying meaning here is

You're welcome to take my help.