Rules Compendium p. 150 (emphasis mine):
A reach weapon is a melee weapon that allows its wielder to strike at
targets that aren’t adjacent. Most reach weapons double the wielder’s
natural reach, allowing the wielder to attack at that reach but not
within its normal reach. A typical Small or Medium wielder of such a
weapon can attack a creature 10 feet away, but not a creature in an
adjacent square. A Large wielder wielding a reach weapon of the
appropriate size can attack a creature 15 or 20 feet away, but not
adjacent creatures or creatures up to 10 feet away. Tiny or smaller
creatures gain no advantage from reach weapons.
Rules Compendium p. 151:
A wielder gains no reach from a reach weapon that is too small. No
additional reach is granted by a reach weapon that is too big.
Based on those two things, we can answer your two questions:
- No. Tiny creatures threaten no squares around them, and reach weapons don't benefit them. As a larger reach weapon doesn't convey any extra bonus, they don't get any reach no matter how big the weapon is. (Though the mental picture of a tiny creature using a gargantuan halberd is pretty funny.)
- If the weapon is inappropriately small, they do not gain a reach boost at all. If it's inappropriately large, they do not gain an extra reach boost, but they would gain the normal reach boost for a creature of their size using a reach weapon. So a Dwarf using a large spiked chain gets the same reach as a Dwarf using a medium spiked chain.
I think your proposed feat is going to work as you intend. A fine spiked chain won't convey any reach, it'd just be like using a melee range weapon only doing a lot less damage.
Specific Q/A
Basic Question: Does a grig (MM 235) wielding a Tiny longspear (PH 116, 119) (2 gp 5 sp; 0.9 lbs.) threaten no squares, adjacent squares,
or squares 10 ft. away?
The grig is tiny, using a reach weapon. It therefore threatens no squares.
Then: What about a grig wielding two-handed a Small spinning sword* (Secrets of Sarlona 137-8) (50 gp; 3 lbs.)?
Threatens no squares. The spinning sword definition says that it's a reach weapon, and tiny creatures do not gain reach from reach weapons.
Then: What about a human wielding as a light weapon a Tiny longspear?
Threatens his natural reach only. Undersized reach weapons convey no reach, but they don't take away your natural reach (since said human could just punch that square).
Then: What about a human wielding two-handed a Large spinning sword (100 gp; 6 lbs.)?
Yes, the human would threaten squares 10' away, and 5' away (as the spinning sword works at close range, like a spiked chain).
Does The Rules Compendium Contradict The SRD/DMG?
No, it doesn't. The SRD's reach definition is as follows (emphasis mine):
Glaives, guisarmes, lances, longspears, ranseurs, spiked chains, and
whips are reach weapons. A reach weapon is a melee weapon that allows
its wielder to strike at targets that aren’t adjacent to him or her.
Most reach weapons double the wielder’s natural reach, meaning that a
typical Small or Medium wielder of such a weapon can attack a creature
10 feet away, but not a creature in an adjacent square. A typical
Large character wielding a reach weapon of the appropriate size can
attack a creature 15 or 20 feet away, but not adjacent creatures or
creatures up to 10 feet away.
Tiny creatures have a natural reach of 0. If you double 0, you get 0. Thus while the RC definition is lengthier, it doesn't contract what the SRD or DMG say.
The only thing in the SRD about oversize/undersize reach weapons is the bit in the above quote about using a reach weapon of the appropriate size. The RC expands on this, but again doesn't contradict what is said. (You could argue that in the SRD, an oversize reach weapon also doesn't grant reach as it's not "appropriate", but they didn't really spell out what they meant so I don't find the RC version problematic.)
Can A Tiny Creature Ever Get Reach?
Yes, as the DMG mentions:
Tiny, Diminutive, and Fine creatures have no natural reach. They must
enter an opponent’s square (and thus be subject to an attack of
opportunity) in order to attack that opponent in melee unless they are
armed with weapons that give them at least 5 feet of reach
That makes it clear that tiny creatures are not forbidden from getting reach, if they have some way to get it. Some examples of ways to do that:
- Shadowstrike (MIC) - On a weapon, activated to give that weapon +5' to it's reach. As it's just a flat addition to reach, it works even if the creature has 0' reach normally.
- Lunging Strike (PHBII) - Feat that allows you to make an attack with +5' to it's reach. Again this is a straight addition, so it works even if you had 0' reach.
- Inhuman Reach (LoM) - Feat that increases your character's natural reach by +5'.
What About Whips?
Whips have a different wording than the other standard reach weapons, as shown here:
The whip is treated as a melee weapon with 15-foot reach, though you
don’t threaten the area into which you can make an attack. In
addition, unlike most other weapons with reach, you can use it against
foes anywhere within your reach (including adjacent foes).
Compare to the Spiked Chain (another reach weapon that works in melee range):
A spiked chain has reach, so you can strike opponents 10 feet away
with it. In addition, unlike most other weapons with reach, it can be
used against an adjacent foe.
The "X has reach" wording on the chain is the same as you see for every reach weapon in the SRD, except the Whip. The first part of my answer covered how those work with larger and smaller creatures. But is a whip special?
- The strictest possible reading says that a whip is treated as a melee weapon. It also happens to have a 15' reach, and that number is specific. Does it change for larger or smaller creatures? The rules don't say. If you wanted to do a very strict reading, it's always 15', no matter the creature size. This starts to make no sense with gargantuan or larger creatures as it actually lowers their natural reach, and would give tiny creatures 15' reach with a whip and 0' reach with any other weapon, so it doesn't work very well at the table.
- A less-strict reading would instead say that a whip is a reach weapon (that can also strike in melee, like a spiked chain) that adds triple to the natural reach, instead of double for normal reach weapons. That means a whip scales up for larger creatures like other reach weapons do (except farther), and tiny creatures get no reach at all when using one. This has the virtue of being consistent with how reach weapons normally work. Also working in the favor of this one is that the reach weapon definition mentions whips as a reach weapon.
So for whips, it comes down to how much stock you put into the odd wording compared to other weapons. I can't give a reference for what the writers actually intended to do there, as one doesn't seem to exist. I do know that the second option is far more consistent in how it plays, so I would certainly favor it in actual play.
The DMG rules are consistent if you count the exact number of spaces, while measuring either way around.
In other words, if you have two creatures adjacent to a middle creature, then count both clockwise and counterclockwise. If the number of spaces (in either direction) equals the number you're trying to count, then there are that many spaces in between.
The trouble here is a geometric property that on a hex grid, Large and Gargantuan creatures are surrounded by an odd number of hexes. If you pick two of these hexes (as possible flanking locations), and count the spaces in between, then you are guaranteed to get different measurements clockwise versus counterclockwise.
However, it is possible to pick two surrounding hexes such that, if you count the in-between spaces both clockwise and counterclockwise, the measurements differ by 1. This is the closest way to split the difference (an odd number), so the DMG uses these measurements to determine flanking. Thus if a creature is surrounded by an odd number of hexes (i.e., they are Large or Gargantuan), then an adjacent attacker can flank with up to two other adjacent attackers.
When two Medium creatures are adjacent to a Gargantuan creature, there are 13 free adjacent spaces. If you can count 6 one way, then you must count 7 the other way. In this case, you counted the short way, but you still counted exactly 6, so they are flanking.
When two Medium creatures are adjacent to a Large creature, there are 7 free adjacent spaces. If you can count 4 one way, then you must count 3 the other way. In this case, you counted the long way, but you still counted exactly 4, so they are flanking.
By comparison, with Medium and Huge creatures, they are surrounded by an even number of hexes. It is possible to pick two of these hexes, such that you can count the same number of spaces in between both clockwise and counterclockwise; these are visibly the flanking positions.
As a side note, the second DMG quote only applies to a square grid. Otherwise you are dealing with creatures that don't have "opposite sides or corners".
So for the purpose of determining whether two attackers flank a Large (or Gargantuan) creature, you will have to count either clockwise or counterclockwise.
In the above examples, there are exactly 4 spaces counted in between the attacking creatures.
Best Answer
The rules for combat in D&D do start to break down in various places when you start dealing with very small or very large creatures and as you've observed, one of these places is flanking.
I'd guess the reason flanking works this way is simplicity. It's a lot easier to say you need to have reach to flank and be on opposite sides than to have special cases around small and large creatures (which would have to take relative size, not absolute size in to account and D&D avoids that wherever possible - just look at how the AC and attack bonuses for size normalise things to be medium). Most of the time the existing rules work just fine.
This is partially because PCs are normally medium or small, monsters are normally a bit bigger than them but not gargantuan and making things more complicated for things that don't come up very much doesn't really add to the game. Additionally, for very large creatures they've already got a sizeable penalty to AC from being so large and normally they have a pretty huge number of hit points. They normally are tough because they can take the hits, rather than because they are hard to hit and because of this don't care much about their AC. A flanking bonus isn't worth anything if you're already hitting reliably without it (unless you're dealing precision damage, but then isn't darting under the gargantuan dragon and stabbing it in the weak point where it's missing a scale just what a rogue should be able to do?).
Then again, if you're running a campaign where this oversimplification will cause problems, possibly because you've got a tiny or diminutive PC or because you're going to be fighting lots of them, you might want to make some changes to bring back some realism.
One easy tweak to make is to say you can flank a creature if you threaten them. If you're two diminutive creatures flanking a tiny creature and you're all in the same square the flanking works fine. This does make it hard to work out who's where in the one square as you still need to be on opposite sides of the creature so maybe scale up the size of everything in a small scale fight. For example, if you've got 3 tiny adventurers fighting 7 diminutive creatures just make the tiny creatures medium for the fight and the diminutive creatures small. The grid rules work fine, you assume people who are fighting are in the same 5ft square so can hit each other (0ft reach) and anyone with a reach weapon still gets the benefit.
If you want to make things a bit more complicated you could say you can only flank someone if you can reach the centre of the creatures space. A medium creature can flank a large creature with a normal weapon but they need a reach weapon like a spear to effectively flank a huge creature. This means you won't flank many larger creatures because you'd need two characters to both have the reach weapons to get a flank. To fix this you could say two characters can flank a creature if their combined reach crosses the creature.
Now two medium creatures with normal weapons can flank a large creature, one of them needs a reach weapon to flank a huge creature and they both need reach weapons to flank a gargantuan creature.
This really penalises rogues and other classes that rely on precision damage as they don't tend to have reach weapons and if they can't flank they won't be able to pull off a sneak attack.
There are already plenty of things in the game that stop rogues having fun (plants, elementals, undead, constructs, and oozes to name a few) so they really don't need any more.As @Carcer points out below, Pathfinder did away with a lot of the immunities to precision damage that various creature types had in 3rd ed so rogues have less things interrupting their fun than they used to. It's worth remembering that Pathfinder made this change for a reason though, and introducing these flanking changes will still unfairly penalise precision damage classes while most likely not changing that much for others.You could fix this by allowing rogues and other precision damage classes to use the old flanking rules to determine if they can get sneak attack damage but still requiring sufficient combined reach to get the +2 bonus.
It does all get rather more complicated though and at the end of the day I'd still say the existing rules, flawed though they are, are probably the easiest and most balanced way to handle things. Just don't play a tiny rogue.