Since you're speaking in generalities, you can use the first person plural pronoun, even if you regard yourself as one of the exceptions (it's assumed that generalities can have exceptions). For example, I might say:
Us men, we never remember our anniversaries.
even if I happen to be a man who always remembers to make a reservation for our anniversary dinner well ahead of time.
I'm talking about men in general, and if I happen to be a man, that's the best way to say it – I should include myself in the first group, since I'm clearly a member of that group.
I suppose I could say:
Those men, they can never remember their anniversaries.
but that seems more fitting for a women to say, not a man. If a man says it that way, he's muddling the conversation, by creating three groups instead of two.
Since these kind of remarks are often intended to be somewhat humorous, we dampen the humor when we get so particular about the pronouns. It's better to be a little self-deprecating and include ourselves with the group, even if we don't exhibit the behavior.
When your friend says:
Indians are the worst drivers, we can take a turn from anywhere!
that's more likely to make me smile or laugh, whle:
Indians are the worst drivers, they can take a turn from anywhere!
sounds more like a peeve or a rant against others in a group that person belongs to, and some of the humor is lost.
In this case I would not use to but there's more to it than that.
When you say come it implies that you're referring to the place you are in. And when you say there it implies that you're referring to somewhere else than where you are.
So neither is really correct but if you switch out come with go, or there with here, it will change, for example.
Are you going to come here?
Which means, are you going to come to the place I am in currently?
Are you going to go there?
Which means, are you going to go to the place where I am not at currently?
But to answer your original question I would not use to.
Best Answer
These are both "correct", after a fashion, but "I don't go anywhere without you" is an unusual phrasing that would only be appropriate in a specific context:
We can assume that, previously, A told B, "Don't go anywhere without me", and so B is simply repeating A's instructions back to him.
Otherwise it would be more natural to say
or, more formally: