It’s quite common.

Banning high-end (and low-end) material is a very common practice. It’s massively more effective and sensible than, say, banning books X, Y, and Z when your goal is to have a certain power level; it gets right to the heart of the issue you’re looking at.

In my experience, however, it’s better to just ban Tier-1 characters. A single level of cleric, while very good, does not make someone a Tier-1 character, but it does enable a lot of other options (Travel Devotion, various Divine Feat options, paying feat taxes efficiently, etc.). Single-level, or few-level, dips in archivist, artificer, druid, or wizard are less frequently desirable, but I can think of arguments for each.

By the same token, it’s pretty common to want players to avoid Tier-5 characters, but sometimes judiciously using a Tier-5 class is appropriate – a couple levels of paladin for Divine Grace, a level of monk for the feats, whatever.

So yes, tell the person playing the cleric that the class is too powerful for the game you’re DMing, and maybe you’ll allow minimal use of it. Tell the player who thinks monk is a good idea that it’s not, and that you need characters to have more power than monk offers in order to be able to make your life easier. A level, to get a punch of relevant feats – fine, if you must. But only if you’re grafting that onto something that could really use it, and be competent.

And it does sound kind of like some of your players are in a place where even allowing minimal usage may cause problems, just because they do seem liable to try to weasel their way into getting more out of you. At least the first time, it may be best to just say “no,” though again I do recommend in general to allow judicious, minimal use of classes outside of the desired tier-band, whatever it is.

Because you are absolutely right – DMing for split-tier parties is extremely difficult and frustrating. In fact, avoiding that situation is exactly the stated purpose of the tier system. See JaronK’s tier list for classes, in the intro spoiler:

Thus, this system is created for the following purposes:

4. To help DMs judge what should be allowed and what shouldn't in their games. It may sound cheesy when the Fighter player wants to be a Half Minotaur Water Orc, but if the rest of his party is Druid, Cloistered Cleric, Archivist, and Artificer, then maybe you should allow that to balance things out. However, if the player is asking to be allowed to be a Venerable White Dragonspawn Dragonwrought Kobold Sorcerer and the rest of the party is a Monk, a Fighter, and a Rogue, maybe you shouldn't let that fly.

^{(emphasis mine)}

The reason anyone cares about tiers at all is to attempt to recognize and address that situation. You are intended to look at the tiers, pick a certain tier or tier-band, and use that as a guide to keeping all PCs on roughly equal footing so that DMing becomes more manageable. So yes, this is “OK,” at least insofar as common wisdom is concerned – it’s actually the whole idea!

• ^{By the way, if your players are looking for acceptable alternatives to favorite classes, this answer may be a helpful jumping-off point.}

The typical approach that I have used and played under is this:

• Tier-4 classes can gestalt with tier-5 classes

• Tier-3 classes can gestalt with tier-6 classes (i.e. non-adept NPC classes)

• Tier-2 and tier-1 classes cannot gestalt

It’s not perfect, but it seems to work pretty well. The higher-tier classes are still more powerful, even with these benefits, but the lower-tier classes are much more interesting than they usually are, giving players more incentive to play them and more reward, in terms of fun, for doing so. My current character is a smorgasbord of monk, paladin, and spellthief, and I’m having a lot more fun than I would as a wizard, even though the guy who is playing a wizard can ultimately do a lot more than I can.

As for distinguishing between classes, the tier list itself covers most classes, and why each class is in its tier explains how to eyeball other classes.

Tier 1 and 2 are relatively easy to adjudicate: if a class has access to something at most levels that trivializes problems in an overpowered or broken way, it’s tier 1 or 2. If it doesn’t, then it’s not. Generally, this means full-casting: a spell progression that eventually hits 9ths. The cleric, druid, and sor/wiz lists have many, many entries that fit this description. The shugenja and wu jen lists are not quite at that level, but probably have enough. And artificer flies here just because his crafting means he can use any and all of the above, if he’s careful.

It’s important to note that tiers should be level- and optimization-agnostic. A broken 20th-level capstone won’t change the tier of an otherwise-fine (or otherwise-awful) class.¹ A particularly broken build that manages to break the game with some shenanigans doesn’t count either.² The idea is that something about the class itself is, at most levels, overpowered.

The difference between tier 1 and tier 2 is, basically, the difference between prepared and spontaneous spellcasting: can the class change what it’s doing regularly, so that it can cover all sorts of very-different situations, or does it get locked into one set of abilities that, while extremely powerful, leave some room for flaws and vulnerabilities?

The other tiers are somewhat fuzzier. Tier 3 is defined as being excellent at one thing and reasonably competent at other things (e.g. warblade), or being quite good at a lot of things (e.g. bard). Tier 4 is decent, or even great at one thing, but useless outside it (e.g. barbarian), or OK at a lot of things but struggles to excel (e.g. rogue). Tier 5 is OK at their specialty, but not even the best at that and utterly useless outside it (e.g. fighter), or is just so spread out that it struggles to get much done (e.g. monk). Tier 6 is NPC classes (and the Complete Warrior samurai, which is basically a fighter with his feats pre-chosen, and chosen poorly). The truenamer doesn’t get a tier, since it doesn’t even work without a lot of work (a high-optimization truenamer probably functions around tier 4 or 5 in a mid-optimization group).

1. _{cf. truenamer, who goes from nigh-unplayable to quite-possibly the most inherently-powerful class in the game when he dings 20th.}

2. _{cf. Pun-pun’s use of ex-paladin}