I didn't do any math, just going off of experience building quite a lot of encounters here. If an exact calculation is what you want, leave a comment and I'll delete this answer.
Coinflip encounter
This encounter will come down to the initiatives rolled. With 13 AC and 28 hit points, neither of the swarms should live more than one attacking round from the party at level 6, even taking into account resistances. They should however take up approximately that round of attack by the players.
Which leaves the Living Spellbook, which gets a practical one round of free attacks because the players will be dealing with the swarms. If they're careless or unlucky (aka, you felt like giving them a hard time), the book will use two multiattacks on the sourcerer or artificer, who should go down fast. At that point the combat depends on whether they were able to dispel the mage armor. If the swarms got off an attack before dying, I'd say those chances are low. Otherwise they are high and the fight should be a breeze for the barbarian and fighter (the arcane explosion should do approximately zero agains them with a Strength saving throw).
So the 2 scenarios I see happening are:
Swarms have relative high initiative
- The swarms attack the artificer and sourcerer for about 50% of their HP
- The swarms die
- The Living Spellbook effectively takes out the artificer and sourcerer out of the fight
- Mage armor isn't dispelled, your barbarian and fighter have a hard time, but should barely win out before any deadly casualties. Bad dice rolls for the player result in a loss
Swarms have relative low initiative
- Swarms die immediately
- Sourcerer dispels mage armor on the Living Spellbook
- Living spellbook becomes a glass cannon with 14 AC, which both the barbarian and fighter easily break
- Fight is a breeze, biggest risk is a small paper cut by one of the twirling paper sheets
Your self-answer covers the standard approach well, so I'll take a stab at an alternate approach.
Dr. Hit-and-Run
Or: How I Learned to Stop Worrying and Love My Mom
Picture this: three frost giants bear down on a party of adventurers. The fighter chops away at the closest one's ankles while the rogue threads a shot between the gaps in their armor. The wizard wiggles her fingers and the giants' eyes glaze over. One swings its axe and barely misses its ally, another stares dumbly at its hands, and the final one flees. The fighter and rogue complain that the wizard should have cast fireball and wish that they had just brought along another rogue.
Wait. That's not what they would say. Without inflicting a single point of damage, the wizard completely changed the course of the fight. Damage is life, control is life insurance.
The Control Barbarian
Most Barbarian Paths focus on increasing personal defense (Bear Totem) or personal offense (Berserker). One, however, focuses on party defense: Ancestral Guardian. At Level 3, they receive the following feature (XGtE, p. 10; emphasis mine):
Ancestral Protectors
Starting when you choose this path at 3rd level, spectral warriors appear when you enter your rage. While you're raging, the first creature you hit with an attack on your turn becomes the target of the warriors, which hinder its attacks. Until the start of your next turn, that target has disadvantage on any attack roll that isn't against you, and when the target hits a creature other than you with an attack, that creature has resistance to the damage dealt by the attack. The effect on the target ends early if your rage ends.
Combat as a control-focused Ancestral Guardian starts out like most barbarians. You rage, run at the biggest monster on the battlefield, and try to hit it in the face. From then on, your allies will scratch their heads and wonder if their barbarian is broken.
Once you hit your target, they will be harassed by a wide array of terrifying creatures; your not-mad-just-disappointed mother, way-too-much-perfume aunt, and have-I-told-you-about-my-time-in-the-war grandfather, will unite to tell your enemy why they should attack you instead of your allies.
Now that your biggest opponent wants you deader than usual, it's time to beat a hasty retreat. The Mobile feat (PHB, p. 168) lets you leave their reach without provoking an opportunity attack. Use it to stroll on over (with +10 speed from Mobile and +10 speed from barbarian's Fast Movement) to some of your smaller enemies and remind them that you're still a barbarian.
Your other allies can now engage the target of your ancestors with near impunity. Even if the harassed creature manages to hit with disadvantage, your allies have resistance to its damage. Congratulations, you just turned your whole party into Bear Totem barbarians who took the Dodge action. If the creature instead decides to chase you (assuming it can compete with your 50' speed), your allies get free opportunity attacks.
The Melee Build
- Basic setup: Rapier and shield. Half-plate brings you up to AC 19, but you have AC 18 in your skivvies, so you might not need it.
- Initial stats: 16 DEX and 16 CON are the only requirements. I'd recommend 12 STR, so you're still a decent grappler with advantage from rage. You could technically dump it, though, if you want.
- Level 3: You get Ancestral Protectors. Without the Mobile feat, this isn't quite awesome yet. You can use a whip's reach to approximate the feat.
- Level 4: Build online. Grab the Mobile feat and run around stabbing anything that moves.
- Level 6: You get Spirit Shield, which prevents even more damage without using any resources.
- Level 8: +2 DEX. Your unarmored AC is now 17+2, so you can ditch the half-plate.
- Level 12: +2 DEX. Your unarmored AC now equals plate!
The Ranged Build
Ben makes a good point in the comments - since Ancestral Protectors works at range, you can make an entirely ranged character if you want.
Instead of a rapier and shield, you just need one hand crossbow. Grab the Crossbow Expert feat (PHB, p. 165) in place of Mobile to get an additional attack (except on rounds when you begin raging). That feat also removes the disadvantage from ranged attacks while you're in melee, allowing you to still run up and tank. Later on, you can grab the Sharpshooter feat (PHB, p. 170), particularly if your party is good at generating advantage. The question requested no multiclassing, but optimally you would dip a level of fighter for the Archery fighting style.
The only thing you really lose is your shield (since you need a free hand to reload the crossbow); in exchange, you gain an additional attack on most combat turns. If you need really long range, you can easily pull out a longbow.
My Experience
One of my current Adventurers League characters is an Ancestral Guardian / Battle Master using a Strength version of this build. When I originally wrote this answer, he was an Ancestral Guardian 7 / Battle Master 3. Since then, he has capped out at Ancestral Guardian 16 / Battle Master 4. He has been quite fun to play and is extremely effective at locking down big hitters.
(This is outside the scope of the question, but don't be afraid to make a strength-based Shadar-kai. Guess what the strength score of my strength-based 10th-level barbarian is? Did you guess 16? Don't tell anyone, but I never raised it!)
How effective is he, really? The short answer is the most convincing: a wizard, protected by ancestors, survives longer than the reckless-attacking barbarian. (You obviously won't be using Reckless Attack, but the numbers are still impressive.) What follows is a condensed version of the math from my answer to "Most effective way of improving survivability for an Ancestral Guardian Barbarian".
The PCs
Class |
AC |
Con |
HP |
Ancestral Guardian 5 |
19 |
+3 |
55 |
Champion Fighter 5 |
20 |
+3 |
49 |
Hunter Ranger 3 / Rogue 2 |
16 |
+2 |
42 |
Evocation Wizard 5 |
15 |
+1 |
27 |
Archfey Warlock 5 |
14 |
+2 |
38 |
The Monsters
This fight will be a Deadly encounter, as it is the boss fight. The PCs will face:
- One CR 5 Gladiator (MM, p. 346)
- Five CR 1 Duergar (MM, p. 122)
A fireball from the evocation wizard will seriously injure or kill several Duergar, leaving the rest for cleanup by the warlock and the ranger/rogue. With that in mind, let's focus on the Gladiator. The Gladiator attacks three times with a +7 to hit, each dealing 11 damage on average (18 on a crit).
The Numbers
As a baseline, the barbarian will last 4.4 rounds against the Gladiator while attacking recklessly, or 6.9 rounds normally.
Class |
Base Rounds |
Ancestors |
|
Ancestral Guardian 5 |
4.4 |
— |
Reckless |
Champion Fighter 5 |
3.4 |
18.4 |
|
Hunter Ranger 3 / Rogue 2 |
2.0 |
7.0 |
|
Evocation Wizard 5 |
1.2 |
3.9 |
No shield |
Archfey Warlock 5 |
1.6 |
4.7 |
|
The wizard can last around 5 rounds if they cast shield on the first two turns of the fight. The fighter survives more than 5 times longer than without ancestors!
Conclusion
Will your dexterity-based Ancestral Guardian barbarian top the DPS charts? Nope. Will your Ancestral Protectors feature draw envious looks from control wizards? Probably. Will you have a blast running laps around enemies while your mom nags them so hard they fail at their one job? Definitely.
Best Answer
Its viability depends on external factors
It is mostly viable, actually the only viable ranged Barbarian build in my opinion1.
The external factors are:
When I played this build, it was quite useful when everything was optimal.
However, its usability decreased significantly with one of these factors missing.
Party composition
As mentioned in the comments of this answer, this tactic is only good if your party members can keep the target of your Ancestral Protectors (Target) away from you.
Either they have to physically hinder its movement, or discourage it with threats of Opportunity Attacks.
You will want to have at least half of the party members to be in melee.
When the Shield Master fighter was not present at our encounters, my Barbarian worked a lot worse.
Available space
If you fight in a dungeon, sometimes you just can't get far enough from the target while maintaining line of sight.
Encounter type
If there are lots of enemies, you might get far enough away from the Target, but his comrades still might be adjacent to you, causing disadvantage on your attacks. You have to hit your target to trigger Ancestral Protectors, so this can be a big problem for you.
Enemy type
If the enemy does not use attacks, only breath weapons or spells, Ancestral Protectors become irrelevant. Unfortunately the resistance is only against his attacks.
Conclusion
It is not viable for dungeon crawling with a party of 4 Sorcerers, but with the average party in the average encounter you will do more than fine.
Recommendation
Use a Hand Crossbow instead of a Longbow and multiclass to Rogue instead of Fighter.
Hand Crossbow
With the Crossbow Expert feat you can attack without disadvantage even if you are adjacent to enemies, negating the worst consequences of a tight space and many smaller enemies.
When you already have the feat, Hand Crossbows give you the most DPR .
Rogue
2 levels of Rogue make Disengage a bonus action, very valuable in tight spots.
Also a Rogue Multiclass provides better DPR increase than a Fighter2, and this area is where you are very much behind a usual Barbarian, as you can't use Reckless Attack and the damage increase during the rage.
1 Being at range gives you extra survivability, by making it harder for others to target and hit you. Ranged characters come away from most fights undamaged in experience, so resistance is mostly wasted on them, and no other significant Barbarian features are usable at range. Other ranged Barbarian's usefullness in combat is below that of a Ranger, and they are famously weak.
2 A single level of Fighter might still be a good idea for Archery fighting style.