[RPG] How to rein in a character’s backstory without discouraging the player

How can I encourage...?

First, though not tagged D&D5e, the answers to these questions contain much valuable advice. I suggest you peruse them:

• How can I encourage role playing in a custom setting?
• How to introduce and encourage role playing in non-roleplayers?
• How do I encourage new players to be more descriptive in their actions and intents?
• As a GM, how do I encourage a party to interact with each other?

However...

There are many different games we play.

^{WARNING: AngryGM liberally salts his excellent advice and analysis with rude and vulgar language.}

It sounds like you want to play the game where "we throw ourselves into someone else's shoes and talk like them."

It doesn't particularly sound like this player wants to play that game.

That's okay.

At any D&D table of \$N\$ players there tend to be at least \$2N+1\$ games being played, in my estimation.^* Not every player equally enjoys each of those games.

The key steps for you to take are to discuss the following with that player, possibly with the entire group participating:

1. I don't see you engaging much in "acting" scenes--do you enjoy that time? (The player may just enjoy watching the show play out.)
2. Do you want to be more active during those scenes?
3. Are there parts of the game you wish we'd be spending more/less time on? (This is a good question to ask all your players, every session.)

And then use the answers to these questions.

Coda: System Matters

I feel I'd be remiss in not adding this: recognize that you're also playing in a system that's not really optimized for that game. Sure, D&D handles it, but it's not built around the sort of roleplay you're describing.

If your player really does want to expand their acting-game, and the things you can try within D&D don't work out, then you may want to consider fantasy-adventure roleplaying games that are built around play-acting.

^{* - note that number snuggles somewhere in between "more games than players" and "not as many games as pair-interactions" for \$N>3\$. \$N=2\$ and \$N=3\$ are special cases.}

The thing done wrong was not accepting the re-roll offer

If the offer is still open, take the sorcerer and wizard players up on their offers to reroll characters if you want this game to go forward.

Why should I do this?

It appears that you were deliberately trying to create some intra-party friction (similar to how Adventurer's League uses Factions for organized play) and that this one mission was bound to create friction based on how you had set it up. Why you needed to get the friction to come to the surface early is unclear, but that's the side effect of the faction quests that you presented.

The other thing done wrong was not perceiving an expectations mismatch.

While I personally like what you were attempting to do insofar as role play is concerned, getting a party to work together with opposed motives is a tricky thing to pull off in this particular game. I've rarely seen it done well (it can be done). I've more often seen it kill off a party and a table.

One of the things you are working against whenever you try to do this is the fundamental structure of D&D 5e as a game. It's built as a team game where a group of people with different skills and talents together overcome challenges, solve puzzles, discover things, and take on adventures.

Most groups of PCs go through four phases: these phases of making an effective team are forming, storming, norming and performing. (Small group dynamics, 101) Your two brothers and three others set up is a classic example of this.

By your having inserted deliberate friction into the team this early on in the campaign, your team of PC's never got past the forming and storming phases. The disagreement on this mission was a reasonable prediction. The offer of "we'll roll up a new character" is as good a solution as you'll get if you want this to move forward.

That your players offered you this solution is a bright shining green traffic light: they recognize the expectations mismatch, and they are offering you a way to keep this campaign alive. Go with their suggestion!

Recommendation

Take them up on their offers! (That wizard and the sorcerer). Have them roll up two new characters so that this party can continue on in their adventures in your world. As DM, put in the work to help them fold their back story into your world so that it aligns with the general theme you already have, and that is mostly acceptable to your players so far.

What could I have done differently?

1. Not create irresolvable character motives in the first place.

   The tension and conflict that you had within the campaign structure may work in writing a book, or a movie or a video game, where the author controls all characters and the narrative. It does not translate well to tabletop RPGs where other people control the protagonists (player characters). The sorcerer was, from your description, more or less maneuvered into a conflict with his new adventuring partners by authorial structural decisions.

   From a design perspective, the mission your sorcerer had would have been better assigned to an NPC whom the players meet/ally with due to something else. This would free up the team to accept, or not accept, participation as a team.

2. The sorcerer's initial mission needed to be more attuned to the general team building that early adventures are meant to establish, rather than in direct conflict with team player goals.

   You mentioned in a comment that you were trying to pull off what Matt Mercer does (apparently) effortlessly. What Matt Mercer and his experienced group of players can pull off doesn't happen overnight. He's been at this for a while, and I'll also point out that the show Critical Role is first and foremost a show meant to entertain. It's not a primer on "How to DM 101." (Though I'd love to play at his table).

3. As a new DM, it's simpler for you to begin a campaign with the basic structure in mind of "a team doing things" and as your players' team forms, storms, norms, and performs, you later introduce the topic of possible differences in player character goals. Crawl, walk, jog, trot, sprint.

4. What you can try to do differently is make sure that you get all of your players to buy into what level of interpersonal tension or friction they can accept before you embark on the first adventure. Unless you get buy in from the players, the "I didn't sign up for this" reaction is predictable.