Short of getting a blue post or twitter reply from a Blizzard dev or rep, the only way to achieve knowledge was testing, which is what I attempted. These are my findings:
General
Massacre is a bonus that you receive when you kill multiple enemies with a short time in between each killing. The difference between Massive Blow and Massacre is that the deaths resulting in Massive Blow have to come from one and the same ability, while Massacre can be a chain of an arbitrary amount of abiliities.
Massacre and other bonuses (such as massive blow, or the one where you have to destroy environmental objects) can overlap without interfering with each other, but they can not be identical. E.g. if you trigger a Massive Blow and then stop doing everything, only the Massive Blow will be registered. However, if you trigger a Massive Blow and then go on two kill another couple of enemies, you will be able to cause an additional Massacre that consists of all kills of the entire sequence.
In my tests, it looked like only attacks and kills help to continue the streak - simply being attacked by monsters wasn't enough and would interrupt the streak.
The minimal amount of monsters which need to be killed in order to trigger the bonus seems to be 10. I have not seen any lower number or at least I can't remember it.
Maximum time between each single kill before the streak is interrupted
This was very hard to test, because it basically came down to doing it a couple of times and guessing. Since I couldn't use a clock to actually measure it (I had to fight monsters), it comes down to my guess: I feel like the maximum time between each kill is around 2 seconds. One second is most definitely too low, and more than two seconds is definitely too high. Knowing Blizzard, though, it's not unlikely that it's some arbitrary number in between, like 1.7850 seconds.
Amount of experience rewarded
Concluding from around 50 or so Massacres it looks like the function is linear, or at least approximately linear. This means that the function will look something like
experience = k*monsters + j
where k and j are constants. In words: If killing 10 mobs gives you 80 bonus xp, then killing 20 will give you roughly 160. The exact amount depends on k and j. j is just the minimum bonus, which I don't know, but guess to be 50 (from memory).
Also, this means that there is no one-to-one correspondence between amount of kills and experience rewarded. You should be able to easily see this for yourself: Killing, for example, 10 monsters will often reward slightly different amounts of experience.
k seems to be based on the levels (relative to your level) and types of the monsters that you're killing. Whether this is because lower-level or lower-health enemies give less XP (making k depend on the experience awarded by the kills), or because Blizzard (rightfully) thinks doing Massacres with lower-level and lower-heath enemies are easier than doing Massacres with higher level enemies, I don't know, but the end-result is the same.
This gives an important conclusion: k does not seem to directly correlate with your level or strength, or the monsters levels or strengths. Instead, it depends only on relative levels or strengths. Killing many monsters with level 59 will still not give you (on average) more experience per Massacre than killing many monsters on level 10, or more precisely, killing weak enemies on level 59 will not give more experience on average than killing weak enemies on level 10.
How the different inputs from different types and amounts of enemies are put together to form k, and what the exact values for the different enemy types are is something that we can't know until Blizzard tells us directly.
To summarize it:
The amount of XP awarded isn't quadratic, exponential, or even something of higher order. This means that the experience awarded won't blow out of proportions if you kill more and more enemies. Instead the amount will rise steadily, making two Massacres with 10 kills each worth about the same as one Massacre with 20 kills (assuming that you're killing the same monsters).
Massacres with "harder" enemies reward more XP, while not being harder to do, since Massacre streaks are not interrupted as long as you're attacking something.
Since evaluating precise values for all monster types, monster levels and monster amounts is not feasible (the number of possible combinations is too high to be accurately tested), it's unrealistic to give some sort of "kills:experience" table.
For the same reason (too many combinations to be sure until we get official word from Blizzard), it's possible that I'm missing some hidden variables which change the amount of experience awarded. However, these, if existent, should be relatively small in magnitude.
Since higher difficulties in normal are designed to be tougher, it can be expected that enemey "strength" relative to you will on average be higher than in normal difficulty. This will most likely result in a higher average k across all battles, meaning that Massacre kill-streak rewards should on average get higher and higher in each difficulty level, although this is just speculation on my part (I haven't tested it yet).
Best Answer
I've paid attention to when this happens, and the only situations I've encountered in which these streaks award no XP is in bossfights or events, where the minions you slay are summoned as a part of the event/bossfight, and award no XP at all.
Since you mention that this has in fact happened to you outside of any kind of event or bossfight, I think it's reasonable to assume that this applies to monsters which award no XP in general.
This shouldn't apply to monsters which are simply much weaker than you, since you can, for example, go back to Act 1 with a level 30 character and still get Massacre and Mighty Blows which award XP just fine.