It would be an improvised weapon.
A special attack such as that would exist in a feat, whether it be core, supplemental, or third party (or a class feature). An example of a feat would be:
Haft Strike
Prerequisite: Two-Weapon Fighting.
Benefit: When wielding a polearm two-handed you may choose to attack with the haft of the weapon. You may only perform this attack as part of a full-attack action. This additional attack is at your highest attack bonus and deals damage like a club of the same size as the pole arm. Although the haft does not possess any of the bonuses associated with the weapon (such as flaming), it does count as magic for the purposes of overcoming damage reduction if the weapon is enchanted. When using this feat, each attack you make in that round (the extra one and the normal ones) suffers a -2 penalty. You only receive half your Strength bonus on damage rolls with this attack. This feat cannot be used with a double weapon.
Normal: The haft of a non-double weapon is considered an improvised weapon and cannot be used as part of a full-attack action.
What you have done, is essentially turned your sword into a metallic club, and would deal damage as a club of its size as an improvised weapon. You are indeed, improvising, in using your sword in such a manner. If a sword was literally designed for such an attack, it would have a pommel on the end of the blade.
DM Ruling
Speak with your DM. He may be agree to lessen the penalty by half or so depending on the role-playing aspects, weapon focus style feats, and etc. Paizo also has Improvised Weapon Feats that can remove the penalty altogether no matter what you want to use as what kind of weapon. He may allow those feats into his campaign.
Key Point
The sword was not designed for such an attack. Just because the wielder improvises, adapts, and overcomes is merely a testament to his knowledge as a martial artist. Doing the best with what you have and making more with less is the improvisational and technological mantras that helped make us who and what we are.
Only a tiny amount of official material even touches on improvised weapons. The feat Area Attack (Savage Species 30) is, quite simply, out of reach for most PCs. A creature that takes the feat Improvised Trip Attack (Dragon #295 73) can make trip attempts by hurling improvised weapons or actual weapons. The feat Improvised Weapon Proficiency (Dragon #295 73) doesn't actually grant it. The prestige class brawler (Dragon #295 72–3) should be awesome with improvised weapons—it does get proficiency with them at level 1—, but it's ultimately a letdown. And, short of taking a standard action to rub onto an improvised weapon oil of magic weapon [trans] (1st-level spell at caster level 1) (Dungeon Master's Guide 230) (50 gp; 0.1 lbs.) and similar substances—which may or may not work depending on the DM—, I don't know of any way to make an improvised weapon magical.
Officially, this is probably just too many hoops for too little benefit
One feat makes using improvised weapons better than using traditional weapons. The general feat Mercurial Strike (Dragon Compendium Volume 1 102) has as its benefit
Any time an opponent provokes an attack of opportunity from you but you are unarmed, you may draw a melee weapon and make your attack of opportunity with it. The target of your attack of opportunity is caught flat-footed for that attack.
Normally, it's impossible to realize this feat's benefit off-turn more than once per round: once a weapon's been drawn due to the feat's benefit, the wielder's no longer unarmed, and dropping a weapon is a free action, and that can't be done off-turn. However, an arrow—therefore a magic arrow—"that hits its target is destroyed; one that misses has a 50% chance of being destroyed or lost" (PH 113) no matter how that arrow is employed (see this question), and that includes an arrow employed as an improvised melee weapon.
C-c-c-combo! (definitely not broken)
The feat Mercurial Strike is the basis of the combination below. It's a mid-level trick for a mundane warrior. This is not a great trick, nor is it even a 100% reliable trick! However, it is a trick that works better when the trick's user employs improvised weapons, specifically ammunition that's destroyed on a hit and can be lost or destroyed on a miss.
Buy the DM pizza then ask the DM if a magic arrow's magical properties apply when a magic arrow is used as an improvised weapon. Mope if the DM still says No.
Get as much sneak attack damage or sudden strike damage as you can.
Take the following feats: Combat Expertise (Player's Handbook 92), Combat Reflexes (PH 92), Dodge (PH 93), Karmic Strike (Complete Warrior 102), the aforementioned Mercurial Strike, and Quick Draw (PH 98). (This can be done by a level 9 human ninja or rogue that has one flaw.) Plus, get a way to threaten your area while unarmed (see below).
As a class skill get the skill Iaijutsu Focus (Oriental Adventure 68–9). (Getting the skill as a class skill might take the feat acquired with a second flaw—I suggest the feat Academy Graduate (Savage Tide Player’s Guide 8).)
Get proficiency with improvised weapons. Several methods are listed here, but, in short, I recommend 1 level of barbarian with the alternative class feature city brawler (Dragon #349 92) then joining the affiliation Bloodbound of the Red Ring (City of Stormreach 99–102) and fighting arena battles until your affiliation score's 11. Those two combined mean you'll suffer no nonproficiency penalty with improvised weapons and you'll have earned the ridiculously metal title of—I kid you not—Slaughter Master! [Cue guitar solo.]
The alternative class feature city brawler also grants a barbarian the bonus feat Improved Unarmed Strike (PH 96-7), which means you now threaten an area while unarmed.
Wade into melee, trying to get hit using the feat Karmic Strike, and, in anticipation, maybe activating an amulet of tears (MIC 70) (2,300 gp; 0 lbs.), a talisman of undying fortitude (MIC 188) (8,000 gp; 0 lbs.), or the feat Stone Power (Tome of Battle 32–3).
When a foe does hit you, employ the benefits of the feats Karmic Strike and Mercurial Strike to make a melee attack with an arrow—or even, if it makes a difference, a magical arrow—against your flat-footed foe, dealing extra damage due to the skill Iaijutsu Focus and sneak attack or sudden strike.
The only melee weapon other than ammunition that I'm aware of that's destroyed on a successful attack is the ocanthus knife (Planar Handbook 77, 78) (450 gp; 1 lb.), which is, obviously, more expensive (and far heavier) than a mundane arrow but—more importantly—the knife does not, like an arrow, have coin-toss's chance of being destroyed or lost on a miss. In this regard, the arrow (or whatever kind of ammunition you prefer) is actually superior to the ocanthus knife: hit or miss, you want that weapon gone so you can make another attack if successfully attacked again, and there's at least a good chance of a dnd-3.5e fragile, elusive arrow disappearing even on a miss.
This trick is incredibly complicated, doesn't work all the time, and is kind of silly, but it's also sort of flavorful: imagine a human barbarian 1/ninja 9 who wades into battle with his bow, shooting arrows willy-nilly, hoping someone will make an attack of opportunity against him so that he can whip out an arrow and eviscerate that foolish foe that thought our hero unable to defend himself! That shtick plus ninja plus freakin' Bloodbound Slaughter Master of the Red Ring? [Cue guitar solo.] That's, like, right out of a 1990s indie comic. The grim-and-gritty adventures practically write themselves.
I know of no other way to make using an improvised weapon superior to using a traditional weapon. Further, I know that even this trick involves stabbing foes with rules-as-written autoannihilating arrows, which will rub some folks the wrong way. What can I say? I'm sorry, but I got nothin' else.
Licensed products' "secret weapons"
AEG's Way of the Open Hand (June 2004) is an officially licensed product bearing the Wizards of the Coast seal. I recommend the book only to those desiring entire runs of games as Hand's editing leaves much to be desired, but the text does include three feats that improve the value of improvised weapons:
The feat Himitsuheiki Novice (75) has easily met prerequisites, and the benefit removes the nonproficiency penalty for improvised weapons, grants a +1 bonus on attack rolls made with improvised weapons, allows improvised weapons to be treated as certain other weapons, and grants proficiency with the club, dagger, greatclub, spear, and whip. A not-horrible feat made more interesting by providing proficiency with both a martial weapon and an exotic weapon.
The feat Himitsuheiki Advanced (76) grants as its benefit, in addition to a bonus when making a feint attempt, the ability to succeed on an easy Spot skill check to take a move action to move up to 10 ft. and simultaneously acquire something that can be used as a weapon; further, if an unattended item can be used as a weapon (like, for instance, if it is a weapon) and it's within your reach at any time during your turn, you can take a free action to scoop it up (which is kind of cool if specializing in disarming, I guess).
The feat Himitsuheiki Master (76) doubles the wielder's threat range with improvised weapons (making the feat Improved Critical (improvised weapon) (PH 95) unnecessary, I guess, if the DM even lets that be a thing) and grants the creature once per round a +2 dodge bonus to AC when the feat's possessor intentionally drops a weapon.
The only other feat barely worth mentioning in licensed products is the don't-blame-me-I-didn't-name-it untyped feat Improvised Weapon Feat (Masque of the Red Death 117), the benefit of which allows the creature to take a move action to simultaneously travel up to 10 ft. and ready for use an improvised weapon extant in the environment… with which the creature automatically becomes proficient. However, Masque (Oct. 2004)—from Arthaus's Ravenloft line—is kind of a separate standalone game despite bearing the Wizards of the Coast-approved seal, so good luck with that. I mean, chances are you were probably already trying the DM's patience by looking through the regular licensed Ravenloft stuff.
Best Answer
Yes.
There is no reason to assume that the quoted statement about arrows breaking pertains only to ranged attacks. If you hit someone with an arrow, it breaks. If you miss, there's a 50% chance that it breaks.