Lacking a more definitive source, I'm working from the metaplot article on Wikipedia, which says, in part, that
The metaplot is the overarching storyline that binds together events in the official continuity of a published role-playing game campaign setting. Major official story events that change the world, or simply move important non-player characters from one place to another, are part of the metaplot for a game.… Because of [some unpopular] events…, many gaming groups choose to ignore the metaplot for a game entirely.
Metaplot information is usually included within gaming products such as rule books and modules as they are released. Major events in the metaplot are often used to explain changes in the rules in between versions of the games….
(I've used the Wikipedia definition here because it was concise, if limited in its application. The TV Tropes's definition is better but more loquacious.)
What I'm struggling with is seeing any upside to a metaplot. By that I mean it seems like a role-playing game's setting will typically be defined by the game's initial release, and if that setting is later altered by the metaplot, future expansions must either accommodate and perhaps (because of previously established expectations) advance that metaplot to satisfy those who are familiar with the now-changed setting and potentially alienate newcomers or ignore the metaplot, leaving fans of the metaplot bereft but making the game easier to grok for newcomers. This sounds to me like a metaplot is a lose-lose proposition.
In short, then, besides disguising mechanical changes behind a narrative façade, why do roleplaying games have metaplots? Has an author or publisher revealed his or its motivation for the inclusion of a metaplot?
I am looking for a definitive answer like As the author of a role-playing game, I opted to include with my game a metaplot because and As a roleplaying game publisher, I prefer my company publish roleplaying games with a metaplot because and similar answers derived from insider knowledge rather than speculation. However, such insider knowledge can be shared by those far distant from how the sausage is made or by those outsiders who are otherwise somehow extremely well informed.
Note: I was considering work once again on my homebrew science-fantasy heartbreaker and was considering a metaplot, but then I realized I had never heard anyone say Wow, the awesome metaplot really ties the setting together! or, pretty much, anything good about a game's metaplot. However, the metaplot concept is so ingrained within the roleplaying game medium—so much so that I was considering it for my own work—, yet, given a metaplot's controversial nature, I don't know why.
Best Answer
Deadlands: A Case Study in Metaplot
Deadlands is a "weird west" roleplaying game that started in 1996 and has had a pretty extensive metaplot, so it's a good case study as to why a metaplot exists. There are two major versions of it: the original Deadlands (often referred to as Deadlands Classic) and the Savage Worlds version Deadlands Reloaded.
To tell the story the author wants to tell
Sometimes, why a metaplot exists is as simple as "beacuse the author wants there to be one". There are a number of instances of this, but a pretty simple one is that Shane Lacy Hensley originally wanted to have a cataclysmic end to the Deadlands setting. When the game became unexpectedly popular, he created "a pretty cool story" to explain how it could still have that ending, yet keep going.
Because fans enjoy it
Deadlands has a lot of characters and a number of stories. In Deadlands Design Diary #2, author Matthew Cutter described one of his missteps was killing off a number of beloved characters in the metaplot and underestimating fan love.
He goes on to say that this was undone in errata, and that this lesson about how much fans cared about the metaplot heavily influenced writing the next major campaign.
To encourage players to buy more books
While I haven't seen a developer outright say that this was a reason, the books often encourage players to buy more books to learn about the "long story" overarching the Deadlands universe. The Deadlands: Player's Guide for Deadlands Classic started off the Chapter 1: Basics telling you that the game was linked by a metaplot spanning three separate RPGs, ending with encouragement to pick them up.
Some of the Deadlands Classic books were more blatant about it and said stuff like "such and such is a mystery, which we will reveal in Supplement Book X". I'll see if I can find one of these books.
To generate excitement for new products
In 2015, Pinnacle ran a Kickstarter for two products: the Stone and a Hard Place plot point campaign, and The Cackler graphic novel. The description for The Cackler especially highlighted the fact that it would change the Weird West forever.
To reward longtime fans
That Kickstarter page for The Cackler challenged longtime fans to guess the secret identity based on clues from previous products.
To provide a story reason for a mechanical change
In Deadlands Reloaded, the Blessed, and their Voodooist variant, were an Arcane Background that worked pretty differently than any other Arcane Background. Pinnacle said in a rules update that they wanted to bring them more in line with the Deadlands spinoff settings and the core Savage Worlds.
While they could have just changed them mechanically by fiat, they decided that the metaplot event "The Cackler" would be a good way to incorporate a story reason to explain these mechanical changes. This-out of-universe explanation was immediately followed by a header and an in-universe reason.
Note that this rules update also contained mechanical changes to the "Harrowed" archetype, but there was no story reason incorporated with those changes.
To tie in with developments with spinoff materials
In the late 1990s, Pinnacle Entertianment Group created a partnership with AEG in which they would create several spinoff games for the Deadlands RPG. The most notable of these was the trading card game Deadlands: Doomtown, which centered on the town of Gomorra (nicknamed "Doomtown").
The RPG fully embraced this with supplement books Doomtown or Bust! and The Collegium. Since that game had its own metaplot, the main RPG metaplot incorporated it.
Compare with: Forgotten Realms novels, Shadowrun video games
To deal with real-world legal issues
AEG and Pinnacle eventually parted ways, and this led to an awkward situation. Pinnacle owned the rights to Deadlands, but Gomorra started in the card game and AEG owned the rights to it and the characters within. Since it was already integrated with the RPG, Pinnacle had to do something to explain its absence. So they said that an unexplained event blew up the town and nobody knew what really happened. The end.
Eventually, the two companies were able to work things out (likely as a result of the partnership from the sequel card game Doomtown: Reloaded), so a new adventure Sawed 'em in Gomorra was created to provide a more satisfactory metaplot resolution.
To make players feel like their actions have long-term consequences
Say that you are playing a Dungeons & Dragons scenario in something like Greyhawk. You kill a big, bad evil wizard who is threatening to destroy the world, but ultimately, your actions are inconsequential. Whether you win or lose, Greyhawk is exactly the same as it was before.
Some settings, such as Torg, Deadlands, and Spycraft have had adventures that had the potential to make a major change in the setting. For instance, Deadlands had an adventure called "Ghost Busters" where a major NPC in the setting was possessed and the players ultimately had to make the decision to kill him, or try to free him. The scenario had a card in the back of the book that you could tear off, where you marked what the fate of that character was and then mailed it in. After a certain date, all of the mailed in responses were tallied and the majority response became "official". The particular fate of that character has been referenced in future settings.
See Has "write in to impact the metaplot" been tried since the 90s? for more information
Deadlands Reloaded has done something similar. They have created four "Plot Point Campaigns" centering around four of the major NPCs in the Weird West, with the promise that each campaign will have a lasting impact on the setting. For instance, the first one The Flood ends with
The benefit to the players of The Flood is that they feel that their campaign had meaning.
Reasons from Other Settings
Forgotten Realms: To stay in line with changes to the core D&D rules
Fourth Edition D&D changed a number of rules systems in general, including how magic behaved and what races were more common. The metaplot for Forgotten Realms had an event called the "Spellplague", which brought the setting more in line with the default 4e setting. Unsurprisingly, the effects of the Spellplague were largely undone by the Second Sundering, which brought the setting more in line with 5e.
Shadowrun: To keep up with real-world changes
Back in the 1980s when Shadowrun was introduced, it seemed logical that the primary way to interact with computers would be to "jack in" to them. Then wi-fi became prevalent in the real world, so this seemed a little silly. While some people have preferred this archaic aesthetic (hence why Shadowrun periodically releases stuff set in the original year of 2050s), others got hung up on the future not having technology that we currently have today.
So the metaplot was changed to include story reasons to make wireless computer access available, and why it was less common in the past.