I'm working on a homebrew revision (and slight buff) of the Sorcerer class in D&D 5e that includes some additional class features. I've read the Unearthed Arcana article about modifying classes, and I've tried to adhere to all the principles it sets forth.
Some of the features I'm adding seem so narratively essential to the class and also so mechanically negligible, at least in the campaigns I've played in so far, that it's easy to figure out where to put them (e.g. the sorcerer can use her body as an Arcane Focus from the get-go).
I can't tell, though, whether a few features I'm considering adhere to or violate the principles in the Unearthed Arcana article. Obviously the safest solution is to rule them out, but I'd like to try to be more thoughtful than that.
How do I figure out: (1) whether they're overpowered/underpowered, and, if they're not, (2) what level they'd be appropriate at?
For example, the sorcerer's Spellcasting class feature currently includes the following text:
Additionally, when you gain a level in this class, you can choose one of the sorcerer spells you know and replace it with another spell from the sorcerer spell list, which also must be of a level for which you have spell slots.
This feels a little too restrictive to me, and I'd like to add the following benefit:
When you finish a long rest, you can choose one of the sorcerer spells you know and replace it with another spell from the sorcerer spell list of the same or lower level.
I feel like this might be inappropriate at early levels, when the Sorcerer knows so few spells, but might be more appropriate later on, when switching out one spell would leave the character's spells known mostly unaffected. But maybe that's totally bonkers and it would be perfect at 1st level, or maybe it would break the game. I've looked at other classes and can't find anything that really "feels" the same, so I'm having trouble using that as a guide.
I've only been playing 5e for a few months so maybe when I have more experience it'll be easier. But in the meantime, what principles can I use (in addition to the ones in the Unearthed Arcana article) to determine whether and at what level a new optional class feature is balanced?
Since posting this, I've found out that the UA playtest class features give almost exactly my example as a feature for the Bard, Ranger, Sorcerer, and Warlock classes at 1st level. The only difference is that the replacement spell has to be of the same level as the spell it replaces, not the same or lower level. Since I was thinking this would be overpowered at level 1, it's possible I've been thinking too restrictively. Either way, however, the question is still relevant for some of the other features I'm considering.
Best Answer
@Thomas Markov's answer is definitely right: playtesting - and playtesting a lot, with a lot of different groups in a lot of different scenarios with a lot of different party compositions - is the only way to be sure.
However, there are a few bits of theory-crafting that can help identify homebrew stuff that may be problematic. And, it's important to remember that "problematic" doesn't necessarily mean "overpowered" - an underwhelming or underpowered option can be just as bad from the player's perspective (though it may be easier to resolve, since the Endowment Effect is less likely to be an issue).
So, let the theory-crafting begin!
Auction
A quick option to ballpark the value of a discrete homebrewed thing is to auction it off to your players. This is much easier with stuff that can be purchased individually rather (eg., with gold or GURPS-style XP) than stuff that comes with D&D-style levels. It's also easier if the currency is valuable; IME, this works better in D&D 3.5 and Pathfinder than in D&D 5e simply because wealth in 5e is (IME) less relevant - there's far less to spend GP on in 5e after about 3rd level (when you've got the weapon and armor you want) except for costly material components.
Assuming that the assumptions can be assumed: auction the thing off to your players.
If it's a thing bought with GP or GURPS-style XP, simply start the bidding a bit lower than the relevant "estimating value" guidelines would indicate (when available) or your own best guess (where such guidelines aren't available). Let the players bid up in reasonable increments, and set the value at whatever the auction's winner bids.
If it's a spell in a Vancian magic system (like D&D/Pathfinder), auction in spell levels. If it's a single class feature in a level-based game (again, like D&D/Pathfinder), auction in levels.
A couple of caveats:
Well, this is broken, part 1
Obviously, a sword of "any creature targeted by an attack from this sword immediately dies, regardless of whether the attack actually hit, how much damage it did, and any abilities the target might have; there is no defense against this effect; in addition, the wielder gets +1000 to all abilities and can cast any spell in the game whenever they want without using an action" is tremendously broken. At the same time, a +1 sword is clearly not broken: the game provides one as possible loot for the players to find.
A feat or class feature that grants +1000 to saves is, similarly, clearly broken; a feat or class feature that grants +1 to saves isn't (eg., Paladins, at 6th level, gain an aura that grants at least that to themselves and their allies within 10 feet).
The question, then, is where the dividing line is. Unfortunately, the dividing line isn't a bright one (if it was, this answer would simply point at that bright line, if the question was even asked). A helpful starting point to determining which side of the line a thing is on is is there any other choice a player would make when this is available?. In answering, leave out that one player everybody knows that keeps making mechanically weak characters because of "flavor" or "backstory" or "incompetence with the system"; system experts only for now.
There's a can of worms here that I'm going to peek into.
Magic Missile is a standard spell for a reason: auto-hitting for decent damage (for its spell level) with a damage type that is rarely (if ever) resisted. Does that make it broken?
In 3.5/Pathfinder, it's relatively common to talk about taking one-or-two level dips in various classes to get at low-level abilities: a 2-level dip in paladin gets you a bonus on your saves equal to your charisma; a level or two dip into fighter gets all of the weapon and armor proficiencies and a couple of combat feats; a dip into rogue gets a bunch of skill points and sneak attack. Are those classes or class abilities broken?
Probably not, but all of these highlight the lack of a bright line between "balanced" and "broken". They can also highlight how the expected duration and theme of a campaign will change character builds: a one-shot-style 5th level adventure means there's no reason to build towards a 6th level ability; dipping into classes for abilities precludes (typically) getting to level 20 in any one class, but if you're not going to get to level 6 anyway, it doesn't matter. Similarly, losing abilities that target undead creatures doesn't matter if the one-shot is built around dealing with golems (and, anti-golem abilities become even more important!).
But, consider a class with a first-level ability "I'm Awesome" that grants a bonus on all d20 rolls equal to the character's number of hit dice (for non-D&D/Pathfinder, read "it's nearly impossible to fail at anything the character tries to do" - compare the GURPS advantage Truly Badass, that scales with the character's power). Is there any reason that any character would choose to not take a level in that class?
Similarly, consider a 9th level spell "Super Wish" that is basically Wish, but without the limitations or risks of something going wrong. Basically, "I super-wish the plot were resolved" without costing anything more than a 9th level spell slot or risking anything. Is there any possible reason that a character capable of casting 9th level spells wouldn't take that spell and cast it every round ("I super-wish that we were on the other side of the chasm and also that I could cast this spell again next round.")?
And, of course, the aforementioned sword of "any creature targeted by an attack from this sword immediately dies, and, and, and...". Is there any possible universe in which a player who can afford that sword won't buy it?
Those are all clearly over-the-top examples, but they illustrate the point: if no sane/reasonable character/player with the ability to gain the thing being homebrewed would choose not to take it, it's almost certainly broken. In some cases, that might just mean that the thing needs to be more expensive - the "I'm Awesome" class ability might be reasonable in some d20 systems at, say, level 15, depending on what the rest of the class offers.
Well, this is broken, part 2
Just as stuff can be obviously too-powerful, they can also be obviously too-weak. The basic question here isn't "is there any other choice a player would make when this is available?", but is is there any player who would choose this when this is available?.
D&D has cursed items that mostly fall nicely into this category: is there any reason a player would choose to use a sword with a -1 penalty to attack and damage (note: in D&D, you always want higher numbers and for your enemies to have lower numbers)? Or a belt that weakens you? Would you ever take a level that grants the ability "I Suck", which gives you a -1 penalty on all d20 rolls?
In all cases, the answer is almost certainly "no".
I say "almost certainly" because it's possible that there's some unknown up-side to the thing (eg., the sword might give you a penalty on attack and damage, but might also ignore any resistance to damage that foes might have or grant extra attacks, offsetting the penalty). Drawbacks definitely play into the balance of a thing.
I should have seen that coming
Players will always surprise GMs, sometimes even with good plans.
Consider a Bag of Devouring: a magic item that looks like it'll let you carry a lot more stuff, but which actually destroys anything that's put into it. One of my first characters was built at around level 12 (in 3.5) and specifically bought a bag of devouring during character creation (with the GM's approval, of course). We later had to get a MacGuffin from somebody who didn't want to give it up; he promised that we could have it if we could get through the door on the other side of a hall. Of course, we couldn't damage the walls or the doors, and couldn't pick the locks on the doors. Water started dripping in through a crack in the ceiling, so I put my bag of devouring under the drip lest the water drown us. As his pet Tempest (a powerful creature of wind and water) was being devoured (since not all of the water got into the bag, it couldn't take a form that would let it try to get out), he grudgingly gave us the MacGuffin.
Consider also a new GM who was trying to build a D&D-like RPG from scratch, who had the misfortune of giving me a sword that could cut through any metal like a hot knife through butter, but could only cut metal (the finest silk would stop it dead in its tracks). Much armor was removed from enemies, many doors were removed from dungeons, and a great many cans were opened before he tried to take it away by forcing us through a portal that would remove the magic from any item brought through it. But: he gave us enough protective stuff to allow one single magic item to be brought through unscathed; of course, the sword was the one item.
Are there "obvious" ways in which the thing being homebrewed can be used in ways that aren't intended? ... was that use of a Bag of Devouring intended?
To be clear: player creativity should usually be rewarded. But, the "off-label" uses should be considered when answering questions about balance. The Bag of Devouring was probably fine: it let us bypass one encounter (and get rid of some "unfortunate evidence" now and then), but it's hard to use it to bypass encounters generally. The anti-metal sword was ... not even "almost", it was broken: too many obstacles were trivially overcome by it, with no real way to limit its utility without obviously targeting that one ability (eg., having everyone in the world suddenly wear a light robe over their armor).
Wait, it can do that too?
This is often a sub-category of "clearly overpowered", but it's worth calling out explicitly.
I had a GM who absolutely loved Marvel comics; as a player, he had a penchant for trying to build Iron Man in D&D 3.5.
As a GM, he gave us a Lantern Ring. At the start, it was just a Ring of Curse Suppression - I had a fighter-type that had an annoying knack for getting blinded; the ring would suppress one curse (which, for this purpose, being blinded counted) active on the wearer. So far, so good, but then he gave us a Lantern with which to recharge the ring. Suddenly, it also granted flight, and a spell a couple of times a day, and another ability or two or three, depending on the wearer (the campaign tied into the 7 Deadly Sins and 7 Capital Virtues; each character's primary sin/virtue affected the abilities granted, much like the various colors of Lantern rings).
The more times a thing's description uses the word "and" when describing its benefits, the more likely it is to be over-powered. Item and/or attunement slots may help mitigate this, but just duct-taping together two items that each require a ring slot and attunement into a new ring that also requires attunement doesn't really address its power level - and it's probably more powerful than you think it is.
Why do these go together?
Having one item that has several uses isn't necessarily a problem: staves basically fall into this category, letting casters cast a few more spells and also functioning as a magic weapon. But, the published staves - usually - have a theme: the Staff of Fire grants access to Burning Hands, Fireball, and Wall of Fire, all fire-themed spells.
Classes, too, are - again, usually - conceptually consistent. That is, their abilities work together to provide an idea of what the archetypical example of that class is. Rogue abilities revolve around skills and subtlety - they can sneak around the battlefield, handle traps, notice what's hidden, etc.. Rangers have a connection to nature, but it's the connection a hunter would have; compare to a druid's connection, which is more of a caretaker of nature.
When building something more than a single item or feat, conceptual cohesion can help reveal whether the whole works together or is just a mish-mash of abilities. A conceptual whole, IME, is less likely to be broken than a mish-mash of abilities. A weak concept is subject to this, too: the aforementioned Lantern Ring had a weak concept ("I love Green Lantern, and Lantern Rings are cool") that pulled in too many disparate abilities.
Let's go back to 3.5/PF-style multi-classing for a moment: with very few exceptions, there is no (RAW) limit to the number of classes a character can have at the same time. So, building a character with two levels of fighter (for some nice bonus feats), a level of rogue (skills, sneak attack), and two levels of paladin (for those bonuses to saves) is totally legal under RAW. There's basically no reason that a homebrew "rogue-adin" class couldn't be built that duplicates that 5th level fighter2/rogue1/paladin2 "rogue-adin"; but, what's the class's concept?
And, would that "rogue-adin" class be overpowered? Maybe; it depends on what they get at 6th level, and 12th, and 20th.
IME, without a conceptual core to build off of, homebrewed sets (which can be "classes" or can be "gear sets" - if you're wearing the Belt, Armor, and Gloves of Some Powerful Person from History, you get their individual bonuses plus some extra thing) want to creep into the overpowered mish-mash realm: "wouldn't it be cool if the class also let you X?" gets harder to answer with "yes, but why would it?" if there isn't a core concept underlying the class.
Mostly, this section is just a word of caution: without a core concept, "this would be neat" can easily turn into "this is overpowered". But, also consider the 3.5 Truenamer (which, per this question's answer, is nigh-unplayable as-written) - there seems to be a core concept there, but the abilities that relate to it simply don't work. Also, consider the PF Kinetisist (which this answer explains, in amazing detail), where the core concept seems to be "I just watched Avatar: The Last Airbender and benders are cool", with a kitchen sink approach to powers that may or may not actually work, either individually or together.
My vampires sparkle
3.5 experimented with non-Vancian magic with psionics. Basically: psions get a bank of power points that they can use to manifest any power they know, with the cost of manifesting a power based on the power's level; most powers can be augmented, costing extra power points to get an enhanced ability (rather like 5e's upcasting).
The problem, for my purposes today, is that psionics were kinda-sorta magic and kinda-sorta not. Specifically, it was left to the GM to determine whether Spell Resistance and Power Resistance were the same thing, and whether anti-magic effects and anti-psionic effects were the same thing. And whether the spellcraft and psicraft skills were the same. And, whether any effect that said "whenever you're the target of a spell" triggered on similar psionic powers (eg., did "whenever you're the target of a charm spell" trigger on the psionic charm powers?).
Without that "psionic/magic transparency", a psion is almost categorically better than a wizard in a campaign that isn't designed around psionics - none of their powers would be affected by spell resistance, and they'd work in anti-magic zones, etc.. There are a few effects that are harder to get, but not terribly many.
If your new thing obviates or nullifies whole categories of abilities, it's probably too powerful. Again, "probably": is becoming immune to fire damage broken?
As a digression: 5e steps away from the assumptions that, in principle, a PC can do anything an NPC can do and vice-versa. Having a homebrewed monster that can nullify a category of abilities is usually fine - look at the number of monsters that are immune to various conditions (or, in 3.5/PF, immune to magic itself, like golems mostly are). Complicating any question about balance is "is this for PCs or NPCs?" - NPCs can get away with abilities that are a lot more powerful than PCs can without breaking the game; I go into more detail here, talking about how abilities can affect Challenge Rating (how powerful an NPC is) vs. (functionally) Effective Character Level (how powerful the PC is relative, theoretically, to other PCs).
Who filed off my serial numbers?
If there's an existing thing that's similar, its cost is probably a good starting point. If there isn't an existing thing that's similar, it's worth asking why the new thing isn't already available - there's a reason "super-wish" isn't an ability.
Potions of cure light wounds are easier to get than potions of cure serious wounds. It's not that the latter are broken, it's just that they're more powerful. Their increased cost is how they're (theoretically) balanced.
9th level spells exist, but they require investing 17+ levels in a class. If your homebrewed ability is roughly equal to a 9th level spell in power and frequency of usability, it probably shouldn't be available at 5th level. At the same time, something with the power of a 1st level spell that's only usable once a day probably shouldn't necessitate 17th level to be available.
Conclusion
All of that said, there is nothing that beats playtesting.
Further: the extent to which any individual point matters will depend heavily on the intended audience of the homebrewed thing - if you're writing a new class for WotC, you need to have playtested it extensively and tried to find the exploitable bits; if you're tossing a neat ring into a dragon's hoard, it's probably enough to look for obviously broken bits; if you're putting together something awesome that your players will get just after finishing the campaign, well, not even the sky's the limit.
And, IME, it's better to err on the side of slightly underpowered and overpriced (eg., if you can't decide whether the new spell should be level 4 or 5, it should be 5). It's easier to make something a bit better or less expensive than to go in the opposite direction.
In case of link rot or not wanting to open a bunch of tabs: