The two sentences mean quite different things.
The one you want (the one that has the same meaning as your original sentence) is the first option:
How many friends were invited by you?
This means that "you" is having a party and invited several people and the asker is inquiring how many people were invited to the party. I will note that this phrasing sounds quite odd to me because it implies that several people are inviting friends to the event and someone is asking how many you, specifically, invited, as if to compare numbers.
A - I hear there's a big party at your house this weekend.
B - Yes. I'm throwing it with two other people. We're expecting about 50 people.
A - How many people were invited by you?
B - Only about 15 of them.
As you can see, this does the work of emphasizing which people B invited himself. This can be done in the active form by emphasizing the word you.
How many friends were you invited by?
The second option means "How many friends invited you [to the party]". As if many people were having an event and you were the target of their invitation. The main time I see this used is when very popular girls in high school-centric films are asking each other how many guys invited them to the prom (a school dance).
A - How many guys have asked you to prom?
B - Five!
A - Wow, I've only had three.
We normally use the passive with verbs that express the idea of agency: something done to {someone or something} by {someone or something else}.
Have, in the sense of "to be in possession of", although transitive, does not involve a doing-to, and for that reason it does not lend itself to passive expression.
On the other hand:
He bought the book
can become
The book was bought by him.
He took the book. The book was taken by him.
Best Answer
If somebody asked you that, the typical response in the passive voice would be: