Learn English – I have not gone. But I *have* gone!

Let's pretend that you recently started working for a company. This company has a yearly Christmas party, and you are wondering whether you're expected to go.

You:

Are we expected to attend these Christmas parties every year?

Coworker:

I have not gone before, and my boss still loves me.

As a native English speaker, there seems to be a frustrating ambiguity. Your coworker could mean:

I have never gone, and my boss still loves me.

Or:

There have been one or more years in which I didn't go, and my boss still loves me.

In English, it seems that "I have gone" doesn't imply having always gone, but "I have not gone" does imply having never gone.

Is there a concise way of indicating the opposite of "I have gone"?

Edit:

By opposite of "I have gone", I meant the opposite of what it conveys.

"I have gone" conveys:

One or more times, I have gone.

The opposite would convey:

One or more times, I have not gone.