Learn English – The only person who/that can read

You can't say with who, not even casually

The word whom is dying out of English, but it's not dead yet. It remains in use in formal speech. In informal speech, people people usually replace it with who except when this sounds especially awkward. Many people aren't sure when to say who and when to say whom, but they recognize some familiar phrases that use whom. Since they're not sure which word to use, they restructure the sentence to avoid having to making the choice. One of those phrases is with whom. These are the same phrases where replacing whom with who sounds especially awkward even in informal speech.

When to say whom

The correct way to use whom is as a relative pronoun introducing a clause where it stands for the object of the verb, which must be a person. The relevant verb is the one inside the relative clause, not the main verb of the surrounding sentence. For example:

We went on a tour of the island with this guide.

This guide is the object of with, so if you make that into a relative clause in a larger sentence, it calls for whom:

This is the guide with whom we went on a tour of the island.

However, in contemporary English, this sounds too formal for casual conversation. So, usually people reword it to avoid using any relative pronoun at all, by moving with to the end of the sentence:

This is the guide we went on a tour of the island with.

When to say who

Who does the same thing as whom, except it serves as the subject of the verb in the relative clause that it introduces. For example:

This guide took us on a tour of the island.

Here, this guide is the subject of the sentence. So, if you make it into a relative clause, you refer to the guide with who:

The is the guide who took us on a tour of the island.

As interrogative pronouns

The same applies when who and whom are used as interrogative pronouns:

With whom did you go on a tour of the island?

Who took you on a tour of the island?

The first question sounds very formal, though, so normally people reword it like this:

Who did you go on a tour of the island with?

This is a typical case where people replace whom with who even though, by the highest standards of formality, it's incorrect.

When replacing whom with who sounds awkward and when it doesn't

Replacing whom with who sounds awkward when whom immediately follows a preposition, as in with whom. Certain stock phrases, like "to whom it may concern", are very familiar, and their echo in people's minds makes who sound wrong in very similar-sounding situations.

There can be no strict rule about this kind of thing, of course, but you can most often get away with saying who in place of whom when who is placed away from the preposition that governs it or when it's the direct object of a verb without a preposition. For example, "This is the guide who we went on a tour with." Whom is still correct in that sentence, but it would sound formal. In formal speech, who would technically be wrong, but many people wouldn't notice. Another situation where who can replace whom informally is in a question with the emphatic or surprised word order, where you repeat a declarative sentence that was just said but with one noun replaced by who; for example, "You kissed who?"

A simple test

If the terminology of grammar confuses you but you have some intuition for when to say he and when to say him, you can tell whether who or whom is correct by restating the relative clause as a full sentence and replacing who/whom with he or him, whichever is appropriate. He corresponds to who and him corresponds to whom.

For example, no one would say:

We went on a tour of the island with he.

Even people who don't have any feeling for when to say whom would say:

We went on a tour of the island with him.

They are correct.

But they carry a lot of emotional and dramatic weight because of the "I am the man who" structure.

Scenario 1: I'm in an elevator, the power goes out for 5 minutes, I use the emergency phone to talk to the building manager, and then everything is ok. When someone mentions it the next week, if I say "I am the man who was in the elevator", I would sound pompous and ridiculous. The more normal expression would be "That was me."

Scenario 2: I'm an assistant on a dig in the Great Pyramid, and am the first person to see a relic that eventually proves something momentous about the ancient Egyptians. 30 years later, I overhear someone disputing that fact. I respond by saying "I am the man who first uncovered the whatsis of Ramses: you, sir, are wrong!" In that case, "I am the man" would be appropriate.

EDIT

Your added example of "I am a man who..." is less dramatic, although still not idiomatic. People are more likely to say "I like to travel" or "I'm the kind of person who likes to travel".