"Whom did she invite?" is grammatically correct, for the reasons you have explained. As Stan and Hunter have explained, this rule is often broken, especially in conversation.
Personally, I cleverly evade the issue in conversation by saying "Who'd she invite?" If a "grammar fiend" (thanks Hunter) attempted to correct me, I would explain to said grammar fiend that "who'd" is a contraction of either "whom did" or "who did" depending on context, and invite him to prove otherwise. I would then walk away and talk to someone else. :)
One more thing. Rather than as you have it, correct is "Whom does this matter concern? It concerns me." Whether a verb can function as transitive or not can be pretty arbitrary in English. If you look up "concern" here: http://www.merriam-webster.com/dictionary/concern you will see that this one is transitive.
Since you're speaking in generalities, you can use the first person plural pronoun, even if you regard yourself as one of the exceptions (it's assumed that generalities can have exceptions). For example, I might say:
Us men, we never remember our anniversaries.
even if I happen to be a man who always remembers to make a reservation for our anniversary dinner well ahead of time.
I'm talking about men in general, and if I happen to be a man, that's the best way to say it – I should include myself in the first group, since I'm clearly a member of that group.
I suppose I could say:
Those men, they can never remember their anniversaries.
but that seems more fitting for a women to say, not a man. If a man says it that way, he's muddling the conversation, by creating three groups instead of two.
Since these kind of remarks are often intended to be somewhat humorous, we dampen the humor when we get so particular about the pronouns. It's better to be a little self-deprecating and include ourselves with the group, even if we don't exhibit the behavior.
When your friend says:
Indians are the worst drivers, we can take a turn from anywhere!
that's more likely to make me smile or laugh, whle:
Indians are the worst drivers, they can take a turn from anywhere!
sounds more like a peeve or a rant against others in a group that person belongs to, and some of the humor is lost.
Best Answer
There is no rule: both are correct and both are used. The object form is more common in speech than the possessive determiner.
As Professor Lawler elsewhere notes:
Notice how he calls the accusative one the normal one: that’s the one with him.