Learn English – question tags with or responses to “Seldom/Rarely/hardly” statements

We have had questions and answers here and elsewhere about how to properly question-tag or respond to normal statements like

I go to the store on Tuesdays
I hate mushrooms

and to "negative-polarity" statements like

I never go to the store on Tuesdays
I don't hate mushrooms.

There are also claims that hardly is a negative-polarity indicator, particularly at To What Extent is Hardly a negative Adverb?

However, as noted at that question, hardly is a strange word, and in particular, I suddenly have cause to wonder how hardly (and its compatriots in strangeness, rarely and seldom) work with tag questions and agreements. Beyond the stated links (and some of the links out from those), however, I have found precious little information on dealing with these words, and precisely nothing on how they affect tag questions and responses.

To my ear, all of these sound wrong as tag questions/responses:

He seldom goes there, ^?doesn't he?
He seldom goes there, ^?does he?

He seldom goes there. ^?—neither do I/Me neither.
He seldom goes there. ^?—so do I/Me too.

He seldom goes there. ^?—But I do.

(You can substitute rarely or hardly ever with the same effect in those examples.)

So, is there a correct tag question or simple response form for these sorts of statements? Is my ear wrong, and some of these formats are accepted? Or is my default reply of "Same here" or a full-sentence counter (e.g. "Oh, I go there pretty regularly") the norm?