Weapon Focus is absolutely a bad feat. In fact, it’s even possible that this was intentional on the part of the system’s designers. It is frequently used as a “tax” for stronger feats or prestige classes.
This is pretty atrocious design, and I can state unequivocally that 99% of those taxed options do not deserve such a tax.
Also, for reference, the entire “chain” of weapon specialization feats are similarly poor. Actually, most of them are a lot worse, requiring Fighter levels and not often being used for prerequisites. The exception is Weapon Supremacy in Player’s Handbook II, which is fairly powerful, but comes way too late and requires way too many things (including Fighter 18).
Your Initial Suggestion
It’s clearly better, but I’m not convinced that makes it good. Ideally, feats should be better than just a straight numerical bonus. The +1 damage is almost meaningless, and +2 to attack is fairly minor as well.
So while better, this isn’t good.
Your Second Suggestion
Better, in that the scaling bonus matters a lot more. Problematic, in that it fundamentally throws off a lot of calculations – attack bonuses already scale much faster than AC, now they’re scaling much faster than they used to. You almost have the flip-side of the problem now: for a lot of builds, Weapon Focus becomes required, because as boring as it is, being without that huge bonus to attack is putting you on a different playing field. That bonus is larger than the difference between medium and full BAB at any given level.
My Suggestion
First of all, I don’t care for the entire concept of Weapon Focus; few characters are going to care very much about the type of weapon they are using most of the time. Most will use weapons all of the same type because they chose that weapon for a reason. And all this accomplishes is giving them extra trouble with damage-type-based DR (e.g. DR 5/Piercing or whatever) that forces them away from their favored weapon.
So my personal preference is to simply eliminate the entire line of feats altogether. Anything that required them in the past, simply loses that requirement. I can imagine that there are cases where I’d want to tax the player, in which case some other feat would take its place.
But if you insist on going this route, I’ want to add features that are actually useful. Not just bonuses, but the ability to use new features of the weapon. This also avoids giving too much of the simple attack bonus, which throws off a lot of calculations.
So something like this:
Weapon Focus
Choose a weapon group from the list below
Prerequisites:
Proficiency in at least one weapon from chosen group, BAB +1
Benefits:
You gain a +1 bonus to your attacks made with weapons from the chosen group that you are proficient with. In addition, each group has its own benefits, listed below. These benefits only apply to attacks made with weapons from the group that you are proficient with.
Axes: You gain the Cleave feat. If you already have it, you gain the Greater Cleave feat. If you already have both, you gain the Whirlwind Attack feat. (Handaxe, Throwing Axe, Battleaxe, Dwarven Waraxe, Dwarven Urgrosh, Orc Double Axe)
Blunt Force Trauma: On a critical hit, your target is Staggered. (Club, Light Mace, Heavy Mace, Morningstar, Light Hammer, Flail, Warhammer, Greatclub, Heavy Flail, Gnome Hooked Hammer)
Bows: You add your Dexterity bonus to your damage. This does not stack with your Strength modifier, if it would otherwise apply: choose the highest of the two. (Light Crossbow, Heavy Crossbow, Sling, Longbow, Shortbow, Composite Longbow, Composite Shortbow, Hand Crossbow, Repeating Light Crossbow, Repeating Heavy Crossbow)
Crossbows: You gain the Rapid Reload feat. (Light Crossbow, Heavy Crossbow, Hand Crossbow)
Double Weapons: You gain the Two-Weapon Fighting feat, but only for these weapons. You may ignore the Dexterity requirements on feats that require Two-Weapon Fighting, but if you do not meet them they function only for weapons in this group. You may apply the enhancements from one end of your weapon to the attacks made with the other end, replacing any enhancements that may have been on that end. (Quarterstaff, Orc Double Axe, Dire Flail, Gnome Hooked Hammer, Two-bladed Sword, Dwarven Urgrosh)
Expert Weapons: You gain the Combat Expertise feat. You may ignore the Intelligence requirements on any feat that requires Combat Expertise. (Unarmed Strike, Sickle, Flail, Heavy Flail, Halberd, Guisarme, Ranseur, Scythe, Kama, Nunchaku, Sai, Bolas, Dire Flail, Gnome Hooked Hammer, Whip)
Flexibility: You ignore Shield bonuses to AC on your attacks. (Flail, Heavy Flail, Whip, Spiked Chain)
Graceful Blades: You may score critical hits even against those creatures that are usually immune to them. (Rapier, Scimitar, Falchion)
Grim Reapers: You may execute a coup de grâce as a Standard Action. (Light Pick, Heavy Pick, Scythe)
Knockout Weapons: An unaware opponent struck with these weapons during a Surprise Round must make a Fortitude save, DC equal to 10 plus the damage dealt by the blow, or be knocked unconscious for 1d4 hours. (Unarmed Strike, Dart, Sap)
Fists of Steel: You may add your Unarmed Strike damage to your attacks made with Gauntlets or Spiked Gauntlets. Any modifiers that apply to both Gauntlet and Unarmed Strike damage (e.g. your Strength modifier) do not stack. (Gauntlet, Spiked Gauntlet, Punching Dagger)
Pallisade: You may set against a charge as an Immediate Action. (Spear, Longspear, Halberd, Trident, Dwarven Urgrosh)
Reach: Your reach with these weapons is continuous out to their farthest extent; you do not have a minimum reach. (Longspear, Glaive, Guisarme, Lance, Ranseur)
Shields: You gain the Improved Shield Bash feat. (Light Shield, Heavy Shield, Light Spiked Shield, Heavy Spiked Shield)
Sicarii: Against flat-footed opponents, you deal additional damage equal to your level. This stacks with Sneak Attack and similar. (Dagger, Punching Dagger, Kukri, Hand Crossbow, Sai, Shuriken)
Spears: On a critical hit, you may choose to leave your spear lodged in your foe’s body. Doing so leaves him entangled, unable to maneuver unless he pulls it out as a Standard Action. Removing the spear without a DC 15 Heal check as a full-round action causes it to deal twice the damage it normally if you’d attacked him with it. (Javelin, Shortspear, Spear, Trident, Siangham)
Spiked Armor: You gain Constrict damage equal to the damage dealt by the Spiked Armor you wear. (Spiked Armor)
Swordsmanship: You gain a +2 bonus to AC, and your bonus to attack from this feat is also +2 instead of +1. If you have Exotic Weapon Proficiency (Bastard Sword), these bonuses increase to +4 each while using one. The AC bonuses do not stack if you use two (or more) of these weapons. (Short Sword, Longsword, Greatsword, Bastard Sword, Two-bladed Sword)
Tactics: A creature tripped by your bolas or entangled in your net are treated as flat-footed against your attacks until they have stood up or disentangled themselves. (Bolas, Net)
Thrown Weapons: You gain the Quick Draw feat. (Club, Dagger, Shortspear, Spear, Light Hammer, Throwing Axe, Trident, Sai)
Unarmed Strike: Your Unarmed Strikes deal damage as if a size category larger, and may be enhanced as if they were a manufactured weapon. You need to be physically present while the enhancements are being worked on, but you do not need to do them yourself. The effective size increase stacks with other effects that increase your size or effective size for the purposes of Unarmed Strikes. (Unarmed Strike)
Viper’s Kiss: You may threaten the area within your reach, and make attacks of opportunity with, whips and whip-daggers. (Whip, Whip Dagger)
Special:
Some of these benefits grant feats: these feats, and any feats that have them as prerequisites, may only be used while wielding one of the weapons from the group that you are proficient in. Any such feats that add bonuses to attacks apply only to attacks made with these weapons; penalties, however, apply to all attacks.
Special:
You may select this feat multiple times, but each time must choose a different weapon group. If a weapon is in two weapon groups, the benefits of both groups apply to its attacks. The bonus to attack rolls do not stack for overlapping weapons.
Special:
A Monk who selects the Fists of Steel group may apply its benefits to any Monk weapon he wields. As an exception, Unarmed Strikes instead deal damage as if his Monk level were four higher than it is. This stacks with other effects (such as a Monk’s Belt) that increases his effective Monk level.
Special:
A Fighter may select this feat as one of his Bonus Feats.
Special:
A Cleric with the War Domain gains this feat for any one weapon group that includes his deity’s favored weapon. For a favored weapon that is in multiple groups, the Cleric may choose which group when he gains the War Domain; he cannot change this choice thereafter.
Analysis
This feat is designed to improve the basic power level of melee in the game. Weapon Focus is a Core feat; lots of other feats and prestige classes require it. Now, it’s a feat most melee probably wants: the bonuses for using a weapon from these groups is pretty significant. I’ve tried to favor the weapons that are inherently weaker with stronger Weapon Focus benefits, but that’s tricky.
Granting a feat as a part of a feat is a weird thing to do, but what this basically ends up doing is providing different requirements for those feats that were previously also very tax-y. Pay two taxes with one bill, I suppose. Things like Rapid Reload (necessary to use crossbows seriously) and Combat Expertise (requirement for several of the Improved combat maneuver feats) are necessary but not that good (crossbow+Rapid Reload still is not as good as a bow, Combat Expertise is a terrible feat on its own). Cleave is the exception; Cleave is a pretty weak feat, but usually it just gets ignored, rather than becoming a tax. On the flip side, axes and swords are the best non-reach weapons, so that balances out (sort of).
A note on Knockout, and the Sicarii: flat-footed is not the same as denied-Dex, and unaware in a surprise round is even more difficult to get. Knockout will, at best, drop one target per encounter, and most encounters you won’t really have that opportunity. Sicarii works a bit more often, but you have to work at it more than Sneak Attack: no simply flanking an enemy and going to town.
A Side Note
In discussing this suggestion in the comments, I was reminded that Legend does similar things with its weapon feats. I do recommend looking to Legend for ideas in that regard: for the purposes of feats, anyway, Legend material can be ported back to 3.5 relatively easily.
While a character that first takes one or more levels in the class gunslinger (or another class that gains the character the class feature grit) is prohibited from taking the feat Amateur Gunslinger, such a character is not prohibited from taking the rogue talent grit. The character won't meet the prerequisite for the feat Amateur Gunslinger that comes from the rogue talent grit, but the character still gets the additional grit feat from the rogue talent.
Note that such a character would still have the feat Amateur Gunslinger, but the character won't realize any of that feat's benefits, and because the feat's source allows no other choices beyond that feat, the feat Amateur Gunslinger isn't eligible for retraining.
If, still later, such character gained another level in a class that granted the class feature grit (like an additional level of gunslinger), the character could immediately trade the feat Amateur Gunslinger for the feat Extra Grit. (This is strictly according to the rules as written. A kind GM may allow a character who already has the class feature grit who takes the rogue talent grit to immediately swap out the feat Amateur Gunslinger for the feat Extra Grit without mandating the character to take another level of, for example, gunslinger.)
Best Answer
So, you're using the epic rules. At those levels, nothing will balance non-spellcasters with spellcasters, and trying to do this is an exercise in futility. That said, the creation of a parallel feat makes some sense and should be doable given our example feat, provided we make the assumption 'one extra quickened spell per round' is balanced and that 'one extra swift-action-y nonspellcaster thing' is of equal value.
When we do so, we see some discrepancies between the Multispell feat and your own.
Prerequisites:
Multispell requires Quicken Spell, which is necessary for its use and furthermore necessary for high-level D&D play between getting 9th level spells and getting Epic Spellcasting. It also requires 9th level spellcasting in a single class, which the game generally treats as equivalent to 'level 17'.
Your feat requires Combat Reflexes, which I happen to like as a feat, but isn't actually central to what your feat does, nor necessarily present in your melee's builds. There's no way for them to definitely be meaningfully contributing at these levels, so we have no idea what feats they've taken. Even if the only classes are melee characters, there's a bunch of different builds that rely on charge+pounce or Tome of Battle classes or whatever, and so don't need Combat Reflexes and are probably too feat-starved to take it.
There are feats that all mundane characters generally have, but the thing is no one takes them willingly cause they are awful and you get them for free. I'm talking about Martial Weapon Proficiency and Simple Weapon Proficiency. Obviously, neither of these is a particularly good feat as a gate against spellcasters that doesn't impact martials, but requiring proficiency in all martial weapons at least stops most full casters and allows most martial classes. Proficiency with all simple weapons lets in the rogue and expert at the cost of also letting in the Cleric (although they could already get in without too much effort if they felt like it) and Adept. There's no way for proficiencies to let in commoners without them spending resources on it or Wizards, Druids, and Sorcerers also being allowed.
@HeyICanChan has provided a pretty elegant solution in requiring two non-simple weapon proficiencies, which blocks all the mentioned full casters barring extra proficiencies from race while letting in both the monk and the rogue. That's probably your best option for a simpleish yet moderately effective anti-caster prerequisite.
Rather than requiring a feat and dealing with the lack of equivalencies detailed above, you might want to just require 'ability usable as a swift action', mimicking only the need for Quicken Spell to use Multispell rather than its status as a feat.
Your feat requires BAB +18. That should really be BAB+17, since it's paralleling 9th level casting. If you want it to be accessible to partial advancement classes like the rogue, bard, and monk or what have you, it should be BAB+12. Note that clerics and druids also get the +12 at 17th level advancement.
Benefit
Blocking just Quicken Spells and for the whole round is kinda weird. Consider instead just banning all spellcasting and spell-like abilities with the extra action it grants. That'd be simpler, and not really any more broken, since we're already in Epic play. The biggest problem I have with your version there is that blanket banning something for the round and involving immediate actions makes it matter what actual initiative count we're on, rather than just order: if I cast a quickened spell on my turn on count 7, I can't take an extra immediate action on count 22 (since that's the same round), but I can on count 3 of the next round (since that's a different round) even though it's still before my next turn, and doing so bans quickened actions in my next turn rather than this one (except then the language is weird).
Another problem with the current wording is that it either allows players to use magic items mimicking quickened spells (which probably isn't what you want) or allows players to use scrolls to enter some prestige classes early, bypassing 'ability to cast X level spells' prerequisites. The latter isn't really a huge problem (I play that way normally), but it might be an issue for you if you aren't used to it.
Special
Yeah, that's the same. It's fine.
In Conclusion
The feat is roughly similar, but could use a little polishing on the prerequisites and the restriction mechanism in the benefit could be reworked for ease of play. It can't ever be actually balanced in the larger sense of the game, but it can be made to thematically and mechanically mirror the multispell feat, but for abilities that aren't spells (and spells with native casting times of 1 swift or immediate action).