Short of getting a blue post or twitter reply from a Blizzard dev or rep, the only way to achieve knowledge was testing, which is what I attempted. These are my findings:
General
Massacre is a bonus that you receive when you kill multiple enemies with a short time in between each killing. The difference between Massive Blow and Massacre is that the deaths resulting in Massive Blow have to come from one and the same ability, while Massacre can be a chain of an arbitrary amount of abiliities.
Massacre and other bonuses (such as massive blow, or the one where you have to destroy environmental objects) can overlap without interfering with each other, but they can not be identical. E.g. if you trigger a Massive Blow and then stop doing everything, only the Massive Blow will be registered. However, if you trigger a Massive Blow and then go on two kill another couple of enemies, you will be able to cause an additional Massacre that consists of all kills of the entire sequence.
In my tests, it looked like only attacks and kills help to continue the streak - simply being attacked by monsters wasn't enough and would interrupt the streak.
The minimal amount of monsters which need to be killed in order to trigger the bonus seems to be 10. I have not seen any lower number or at least I can't remember it.
Maximum time between each single kill before the streak is interrupted
This was very hard to test, because it basically came down to doing it a couple of times and guessing. Since I couldn't use a clock to actually measure it (I had to fight monsters), it comes down to my guess: I feel like the maximum time between each kill is around 2 seconds. One second is most definitely too low, and more than two seconds is definitely too high. Knowing Blizzard, though, it's not unlikely that it's some arbitrary number in between, like 1.7850 seconds.
Amount of experience rewarded
Concluding from around 50 or so Massacres it looks like the function is linear, or at least approximately linear. This means that the function will look something like
experience = k*monsters + j
where k and j are constants. In words: If killing 10 mobs gives you 80 bonus xp, then killing 20 will give you roughly 160. The exact amount depends on k and j. j is just the minimum bonus, which I don't know, but guess to be 50 (from memory).
Also, this means that there is no one-to-one correspondence between amount of kills and experience rewarded. You should be able to easily see this for yourself: Killing, for example, 10 monsters will often reward slightly different amounts of experience.
k seems to be based on the levels (relative to your level) and types of the monsters that you're killing. Whether this is because lower-level or lower-health enemies give less XP (making k depend on the experience awarded by the kills), or because Blizzard (rightfully) thinks doing Massacres with lower-level and lower-heath enemies are easier than doing Massacres with higher level enemies, I don't know, but the end-result is the same.
This gives an important conclusion: k does not seem to directly correlate with your level or strength, or the monsters levels or strengths. Instead, it depends only on relative levels or strengths. Killing many monsters with level 59 will still not give you (on average) more experience per Massacre than killing many monsters on level 10, or more precisely, killing weak enemies on level 59 will not give more experience on average than killing weak enemies on level 10.
How the different inputs from different types and amounts of enemies are put together to form k, and what the exact values for the different enemy types are is something that we can't know until Blizzard tells us directly.
To summarize it:
The amount of XP awarded isn't quadratic, exponential, or even something of higher order. This means that the experience awarded won't blow out of proportions if you kill more and more enemies. Instead the amount will rise steadily, making two Massacres with 10 kills each worth about the same as one Massacre with 20 kills (assuming that you're killing the same monsters).
Massacres with "harder" enemies reward more XP, while not being harder to do, since Massacre streaks are not interrupted as long as you're attacking something.
Since evaluating precise values for all monster types, monster levels and monster amounts is not feasible (the number of possible combinations is too high to be accurately tested), it's unrealistic to give some sort of "kills:experience" table.
For the same reason (too many combinations to be sure until we get official word from Blizzard), it's possible that I'm missing some hidden variables which change the amount of experience awarded. However, these, if existent, should be relatively small in magnitude.
Since higher difficulties in normal are designed to be tougher, it can be expected that enemey "strength" relative to you will on average be higher than in normal difficulty. This will most likely result in a higher average k across all battles, meaning that Massacre kill-streak rewards should on average get higher and higher in each difficulty level, although this is just speculation on my part (I haven't tested it yet).
In general, from about Normal difficulty Act 2 onwards, a +10% or higher bonus (from putting a Ruby into a socketed helmet) is going to beat the pants off of any +XP bonuses you can find.
The +XP bonus enchant tends to max out around +15 XP, which would mean it would beat a +10% XP enchant if monsters were worth less than 150 XP per kill. About midway through Act 2, I sampled some of the monsters XP values as I was killing them, by monitoring my current XP to level before and after encounters. Routinely, the average per-kill was more than 200 XP. As you progress through the game, the amount of XP per-kill will increase, and easily dwarf the small bonuses you can get from +XP gear.
This doesn't mean that you should not use +XP enchants on other equipment - but especially later in the game, it becomes less and less worth the investment. Almost always, if you have a choice between a socketed helm with a good quality ruby, or a non-socketed helm with a +XP enchant, take the helm with the ruby.
Bear in mind that the XP you gain from monsters also depends on your level versus the level of the monsters, and information about monster levels is not readily available in-game. Thus, the only real way you can say how much the monsters you are fighting are contributing to your XP is to hover over your XP bar before and after killing them, and then do the math based on these values.
Best Answer
It works in multiplayer exactly the same way it works in single player. If you are within the "screen and a half" range for getting xp, it works as if you had killed the monster yourself.
I played through all of normal difficulty in a party of 3, and the person who had the most +xp per kill was always the one who leveled first. Most of the time it was me, and I wasn't getting most of the killing blows based on the number of monster streaks that the other two were reporting.