Some background: I love to homebrew things. I'm constantly creating new stuff for fun. These are the most important things I've learned in the process.
Compare it to official race/classes
The first (and most important) step is to compare it to official and published race/classes. Usually the PHB is enough. You can either create a rating system of your own or use a well acepted rating system from the internet, e.g. This homebrewing race guide. Note that it only works if you assume the original devs did a decent job in balancing whatever they published. For example, in my opinion, this is not true for feats in 5e, which are heavily disbalanced. It works for races and classes though.
Usually, you should be creating something "close" to the original features. Do you have something that deals elemental damage in an area? Compare it to Dragonborn's Breath. Do you have a flying race? Compare it to Aarakocra. If you stray too much from the original books, you might find yourself in a difficult position where only experienced DMs should be. Do you have a heavily martial type class? Compare it to Barbarians or Fighters, etc.
The Wizard's official site also has some insights on Modifying Classes and the DMG has some insight on modifying races. Although the Unearthed Arcana (UA) is not playtested and well balanced, it might serve as a good start as well. Some of the UA races/classes have feedback from the community. Check them.
Playtest it
You choose how. Find a table of friends that are fine with you using them as lab rats. Search for an online playtest group. Do whatever you are able to. But most importantly, playtest it. Nobody has crystal balls. Nobody knows if something is going to be balanced or not. The official release of 5e had lots of playtests before actual release. Sorry, but you're not better than an entire devs' team dedicated solely to game developing and balancing. You will screw things up and make OP and UP homebrews that will completely break the game or suck, if you try enough times. That's OK. That's why you try it out: to check if it's okay. If it is not, remake it taking into account whatever you learned from your tests.
Compare it to your previous experiences
Well, the whole point of playtesting is to learn. It is okay to make mistakes. But you need to learn from it. Oh, 10d6 was too much damage for a race feature that you get at 1st level? Who would have thought?! Okay, let's tone it down to 1d6 and try again. Oh, I'm trying to create a flying race. Hmm, Aarakocra and that Winged Elf that I tried out seemed OP for Tier 1, maybe I should give the wings only at 5th level? That might be better. Anyway, learn from your past mistakes and even from published mistakes.
Get Feedback
After you have homebrew'd your thing, you can always post it here and get some advice, or, if you feel more like a idea generation and feedback process, on usual RPG discussion forums. Obviously, the feedback from people that actually played with that (i.e. the people which you playtested with) are the most important ones, but the internet has some good theory crafters around.
Bonus
And honestly, that's all I can say. Other than that, you might find specific advice if you google your problem or query search here at RPG.SE. For example, if you google "Homebrew race guide 5e" you will find the guide I linked to. If you google "Homebrew class guide 5e" you will find some decent guidelines for classes (granted, balancing classes is way harder than races, reason this question might get closed).
Disadvantages vs Advantages
This edition, opposed to, for example, 3.5e, went for a more "only bonus" path. Essentially, this means you don't lose things by getting something new. You don't lose your race bonuses when you choose a subrace. Most feats are positive-only. The few trade-off things, like Sharpshooter, are optional (i.e. it's not a permanent debuff on attack bonus). You don't lose ASI with most playable races (exception for Monstrous Races in VGM, which are explicitly stated as not balanced and that you should take care using them). Races have resistances, not vulnerabilities. Classes have, at most, restrictions when they can use their features, not actual impairments, the only exception I'm aware being the Metal restriction for Druids, which is not exactly huge (in fact, it has been stated in the Compendium that this restriction is thematic and linked to D&D history, rather than balance, and removing this restriction is ''not going to break anything in the game system'').
This is a way of thinking seen in many online games, where the devs prefer to buff things rather than nerf. Basically, we feel happier getting better things. We actually feel happier getting a +1 rather than a +7/-6.
By this philosophy alone, trying to balance a huge buff with another huge nerf is usually a bad idea in this edition. If that's not enough for you (and shouldn't be), as I mention in this meta, most features are situational. When you try to balance a huge bonus with a huge impairment, there are two most likely scenarios, both bad for everyone:
- The huge bonus is not that huge and the impairment is too restrictive. As an example, think about the flying example: Having wings is huge in early levels. So huge that it's banned from AL. If you try to balance that, with, let's say, (literally 30 secs of thinking here, don't judge) vulnerability to piercing (so arrows will wreck it), when you're inside a cave where you can't fly, all you have is vulnerability to piercing, which will probably mean you get instakilled.
- The impairment is not that bad or can be exploited in some way, so all you have is a huge bonus. Same example as before, but there are no/few piercing attacks in your campaign. All you have now is a flying race. Sure, physical damages are common and this is unlikely, but might not be that unlikely for, e.g., poison resistance/immunity.
Best Answer
The official texts encourage it.
The official books, especially the DMG, explicitly encourage homebrewing. While the other answers give some nice perspectives, it's worth pointing out that it's offically sanctioned too.
I'm citing from my edition of choice, 5e, but other editions also encourage homebrewing.
To address your specific concerns, DMG 9 recognizes that the world will become your own even if you use a premade setting:
...and DMG 285 has a section on modifying or making your own player options:
Similar text exists in the official books for pretty much every feature of the game, ranging from spells to magic items to backgrounds.
In fact, the DMG gives you full authority over the game, even over the rules (263):
There's even some articles online that explains the developer intent behind the classes, and gives some insights into how you might change them if you wanted:
3.5e, the only other edition I have access to, also has similar guidance. Briefly, the table of contents for the first DMG has sections for "Modifying a Common Race" on pg 171 and "Creating New Classes" on pg 175. The second DMG has an entire section on making your own prestige classes, and indeed assumes that prestige classes are usually created by the DM (203) (though it didn't really play out that way):