The Greek word for speech being ομιλία (omilia), I would suggest something along the lines of omilophobia.
As aronisstav mentions in his answer, there are good reasons to leave the i in. It means the word refers to fear of speeches (plural) and avoids up to some level confusion with other possible interpretations, like the fear of speaking oneself. The word would then be omiliophobia.
If you want to use a word that is in use already, although it does not seem to be very widespread, there is homilophobia, defined on Definition-Of.com as : a fear of sermons. It also appears in the list on phobialist.com.
The aspiration (the h) has been dropped in modern Greek, and the modern word does not just mean sermon.
Take your pick, with an h or without, with the i or without:
Homilophobia: it exists, it is in use, it derives from classical Greek. One setback is that it is up to now used to mean "fear of sermons" rather than all speeches.
Omilophobia: it is new, it does not derive from classical Greek, but it is close enough to the existing version that it should be understood. It does not just refer to sermons.
Omiliophobia: also not from classical Greek, but grammatically the “cleanest” formation. It specifically refers to hearing speeches, but it also is furthest removed form the existing version.
Some concerns have been raised in the comments and I will try to address them.
How do we distinguish between the fear of giving speeches and the fear of listening to them?
As aronisstav explains, the version omiliophobia would deal with this confusion from a grammatical point of view. Arguably that is not the strongest possible way since very few people will have enough of a grasp of (modern) Greek to appreciate the distinction.
However, there actually is a distinct word for the fear of giving speeches: glossophobia (from The Free Dictionary), which is recognized enough to have a site for sufferers, Glossophobia.com.
Since there are actually two different words for the fears of giving and hearing speeches, the distinction in meaning should be clear.
Credit for the glossophobia addition goes to marantou who posted this as an answer to this question before realising the opposite was asked.
Nobody will recognize the word, it doesn't exist, it is not used or known!
Well, homilophobia actually does exist, in the sense that it is used, there are several sources for it. The two proposed variants without the h are based on modern Greek rather than classical Greek, and indeed they do not exist. However, they are close enough to the existing version that they should be recognizable.
As to the word(s) not being well-known — I think that is quite obvious. If the word were well known, chances are this question would never have been asked. If the most important goal would be for the message to be understood, there would be the simple option of using the description as used in the OP. Rather, the question was for a single word to describe the same.
I would like to encourage those that are convinced a well-known single word should be used for the described fear to post that word as an answer. A suggestion like logophobia has the specific disadvantage that people will recognize it — and ascribe the another meaning to it than what is asked for. It means a fear of words (far removed in meaning from speeches) and I assume some people at least would even make a link to “fear of logic”. I believe it is better to make people look up a word or to have to explain it, than to have them assume they understand it — when they actually understand something you are not trying to convey.
Best Answer
Invented-Greek epistemonophobia (επιστημονοφοβία epistēmonophobia, pronounced "e.pis.tee.mo.no.pho.bi.a"), from Modern Greek επιστήμονας epistēmonas "scientist"
Or specifically epistemonissophobia (επιστημόνισσοφοβία epistēmonissophobia), from επιστημόνισσα epistimónissa "female scientist".
Or generically epistemophobia (επιστημοφοβία epistēmophobia), from Ancient Greek ἐπιστήμη epistḗmē (pronounced "e.pis.tee.mo.pho.bi.a") "science, knowledge", from ἐπίσταμαι epístamai "to know, to have the knowledge of something".