[RPG] Limit to number of weapon/armor enhancements

The DMG states on p288 under Adding New Abilities:

A creator can add new magical abilities to a magic item with no restrictions. The cost to do this is the same as if the item was not magical. Thus, a +1 longsword can be made into a +2 vorpal longsword, with the cost to create it being equal to that of a +2 vorpal sword minus the cost of a +1 sword. If the item is one that occupies a specific place on a character’s body (see Magic Items on the Body, page 214), the cost of adding any additional ability to that item increases by 50%. For example, if a character adds the power to confer invisibility to her ring of protection +2, the cost of adding this ability is the same as for creating a ring of invisibility multiplied by 1.5."

So you can enchant your spellbook as you like (unless the spell description specifically restricts it). Other items if they take up a body slot incur the 1.5x extra cost. It goes without saying that the DM has the final say on what can and can't be done in their game world.

You cannot wear armor that is not sized for you

Most items are allowed to work when missized, and weapons work with a penalty, but no such rule exists for armor, and armor is referred to as having a particular, fixed size.

Armor spikes don’t necessarily have to be the same size as the armor they’re on

There simply isn’t a rule anywhere that says so.

At first glance, this seems absurd and an oversight, but I think a case can be made for “extra large” spikes. This is most notable in the case of armor made for half-giants and goliaths, as their Powerful Build means they are somewhere halfway between Medium and Large, and can use Large weapons without penalty though they wear Medium armor. For a goliath, it makes sense that they would wear Medium armor with Large armor spikes.

The handedness of armor spikes is undefined

Armor spikes are supposed to be Light weapons, do not get used in any hands, and at no point do the rules address over-large spikes directly. In theory, Large armor spikes on a Medium creature would now be one-handed weapons and thus eligible for wielding “in two hands,” but there is no way to actually use two hands while wielding them. Thus, I do not think it is possible to get the 1½ Str to damage or the 2:1 Power Attack ratio with this method. They do get the Str to damage and 1:1 ratio of a one-handed weapon, because that is what they are. The rules do not actually say that one-handed weapons get those while “wielded in one hand,” unlike the two-handed versions.

But you could take things a step further: Huge armor spikes on a Medium creature are two-handed weapons. As with one-handed weapons in one hand, two-handed weapons don’t actually specify that you only get the improved damage when wielding them in two hands, because that’s assumed. The text is “two-handed weapons and one-handed weapons wielded in two hands.” So in this case you would get improved Str to damage and Power Attack. And a −8 penalty to attack for wielding a ludicrously oversized weapon.

The armor still does not take up any hands, in any of these cases.

This really does not seem like the right way to go

You have a monster that’s given a higher CR than it probably deserves, against PCs nearly twice that CR in the first place. The sensible thing to do here would be to houserule them into something better, or at the least give them class levels (I recommend Fighter 6 with the Dungeoncrasher ACF from Dungeonscape, or else Crusader or Warblade). This particular collision of the rules is wonky, and still will not provide particularly effective combatants.