Phrasal Verbs – ‘Hand it Over’ vs ‘Give it to Me’

(A small issue that doesn't relate to the grammaticality: it seems unnatural to me to use a defining relative clause after a proper noun, at least in this particular sentence. So I'm going to use commas before the relative clauses in all of the sentences that I discuss.)

Like any other clause, a relative clause generally needs a subject. In your sentence (2), "I gave John the book," "I" is the subject. If it were possible to form a relative clause with the same grammatical structure as sentence (2), it would be as follows:

3. ?He saw John, whom I gave the book.

This sounds (barely) grammatical to me, but I expect many other people would find it ungrammatical. I don't think a native speaker would ever prefer this over an alternative formed from "I gave the book to John":

4. He saw John, to whom I gave the book.
5. He saw John, who(m) I gave the book to.

Sentence (3) might be a marginal case like "Whom did you give the book?": as far as I could determine, all native speakers agree that the preceding sentence sounds bad, but there is disagreement about whether it is outright ungrammatical.

It took some searching through a number of examples of "to whom I gave [direct object]" for me to find one example of "whom I gave [direct object]" in Google Books:

Next morning I found many men and women dead, whom I gave water last night. (Hiroshima, by John Hersey)

Passive-voice versions of ditransitive verbs

In your proposed sentence "He saw John whom the book was given," you've removed the original subject of the sentence, the pronoun "I." But as I mentioned previously, the relative clause is required to have a subject. The subject can't be "whom," as that is an object pronoun. The only remaining noun phrase is "the book." This is an object in your sentence (2). So your proposed sentence would have to use a passive voice structure, where the relative clause corresponds to an independent sentence like the following:

6. *The book was given John.

This kind of passive voice is disfavored in present-day English: it sounds archaic at best, and just wrong at worst.

In general, people only use the following passive forms for ditransitive verbs:

7. The book was given to John.
8. John was given the book.

So it would be grammatical to say either of the following:

9. He saw John, to whom the book was given.
10. He saw John, who was given the book.

Intuitively, I feel like the addition of "over" implies that the destination is a house. I can think of a few phrases that fit this description:

Several friends came over [to my house] for dinner.

Jake brought his baseball cards over [to his friend's house].

Sandy's friends brought sleeping bags and slept over [at Sandy's house].

I can't think of an example where you would use one of these without implying that the destination is a house. For example, you wouldn't really say "I'm at work, do you want to come over and pick me up?" Or, "I brought all my books over to school."

I know this isn't a very scientific answer and some will disagree, but as a native speaker I feel like that's the shade of meaning that "over" gives in this case.