Learn English – Active Passive Voice

John Lawler's comment sums it up:

Technically, the teacher is wrong. The teacher is confusing question formation, pied-piping, case marking, preposition stranding, and formality levels. The sentence produced was the passive of the sentence given. There are other rules beside passive involved, however. All of the following are grammatical:

• The vase was broken by who?
• The vase was broken by whom?
• Who was the vase broken by?
• Whom was the vase broken by?
• By whom was the vase broken?

They are not always appropriate for the same circumstances, like any different sentence.

Here's a paper that describes some relevant phenomena: "On the Grammatical Status of PP-Pied-Piping in English: Results from Sentence-Rating Experiments," by Seth Cable and Jesse A. Harris.

In English, the "neutral" position for question words is generally at the start of the sentence. Many grammatical analyses treat this as the result of a process, "wh-movement," that moves these words from another position.

Cable and Harris give the following example:

(1) Simple Wh-Movement in English
a. She left John
b. Who₁ did she leave t₁ ?

It's also grammatical to leave a question word in place ("wh-in-situ"), as in "She left who?", but in standard English this kind of structure is not the default: it's generally less appropriate than wh-movement except in certain circumstances (e.g. echoing a declarative sentence to express surprise). This is a valid stylistic reason why someone might object to your daughter's sentence. The teacher was wrong to call it "ungrammatical."

Sometimes, other words can also "move" with the question word. The term used for this is "pied-piping," since the idea is that the question word metaphorically abducts other words a bit like the Pied Piper of Hamelin abducted children.

This phenomenon is quite interesting to linguists so there has been a lot of study of it. When the question word is the object of a prepositional phrase (PP), it's generally optional to move the preposition to the front of the sentence along with the question word. Cable and Harris's explanation:

(2) Pied-Piping in English
a. She left that guy.
b. [ Which guy ]₁ did she leave t₁ ?

Interestingly, when a wh-element is complement to a preposition, there appears to be some optionality in whether the preposition is ‘pied-piped’ when the wh-element undergoes movement. That is, it would appear that English freely permits both the structures in (3). In sentence (3a), the wh-word has not pied-piped the higher PP. Such structures are commonly referred to as ‘preposition stranding’ or ‘P-stranding’. In sentence (3b), the wh-word has pied-piped the PP, a structure referred to as ‘PP-pied-piping’.

(3) The Optionality of Pied-Piping PP in English
a. Who₁ did she leave [_PP with t₁ ]?
b. [_PP With whom ]₁ did she leave t₁ ?

Although both the structures in (3) are commonly reported in the syntactic literature as ‘acceptable’, the grammatical status of PP-pied-piping structures is somewhat unclear, especially when compared to their preposition stranding counterparts. While speakers recognize structures like (3b) as English, such structures are not particularly colloquial. It is sometimes said that such structures are limited to particular registers, but often what is meant by ‘register’ in this context is unclear. After all, structures like (3b) are no longer a regular occurrence in formal written English either.

So the teacher's sentence "Whom was the vase broken by?" is not ungrammatical either.

However, it may sound funny because in general, the contexts where people use "whom" are also contexts where most people try to avoid preposition-stranding. (This point is also made by the answer to the following question: "Prepositions at the end of sentence and whom".) This would be a valid stylistic reason to object to the teacher's sentence.

As other people have mentioned, this consideration means that the following sentence might be the best, stylistically speaking:

• By whom was the vase broken?

However, a valid stylistic objection to this sentence is that to most people it sounds highly formal, old-fashioned, or pretentious. Most people would say, and many people would write

• Who was the vase broken by?

People could object that this sounds too informal. I think that's silly though, so I won't call that a valid stylistic objection. Using "who" in this position (not "whom") and stranding prepositions are both generally considered acceptable by educated people. But if the teacher is a pedant or "stickler," that might not matter.