It would be an improvised weapon.
A special attack such as that would exist in a feat, whether it be core, supplemental, or third party (or a class feature). An example of a feat would be:
Haft Strike
Prerequisite: Two-Weapon Fighting.
Benefit: When wielding a polearm two-handed you may choose to attack with the haft of the weapon. You may only perform this attack as part of a full-attack action. This additional attack is at your highest attack bonus and deals damage like a club of the same size as the pole arm. Although the haft does not possess any of the bonuses associated with the weapon (such as flaming), it does count as magic for the purposes of overcoming damage reduction if the weapon is enchanted. When using this feat, each attack you make in that round (the extra one and the normal ones) suffers a -2 penalty. You only receive half your Strength bonus on damage rolls with this attack. This feat cannot be used with a double weapon.
Normal: The haft of a non-double weapon is considered an improvised weapon and cannot be used as part of a full-attack action.
What you have done, is essentially turned your sword into a metallic club, and would deal damage as a club of its size as an improvised weapon. You are indeed, improvising, in using your sword in such a manner. If a sword was literally designed for such an attack, it would have a pommel on the end of the blade.
DM Ruling
Speak with your DM. He may be agree to lessen the penalty by half or so depending on the role-playing aspects, weapon focus style feats, and etc. Paizo also has Improvised Weapon Feats that can remove the penalty altogether no matter what you want to use as what kind of weapon. He may allow those feats into his campaign.
Key Point
The sword was not designed for such an attack. Just because the wielder improvises, adapts, and overcomes is merely a testament to his knowledge as a martial artist. Doing the best with what you have and making more with less is the improvisational and technological mantras that helped make us who and what we are.
D&D 3.5e isn't historically representative.
If you look at the picture of the swords in the Player's Handbook, you will find that the longsword is actually a picture of a bastard sword, the bastard sword is actually a picture of a longsword and the greatsword is actually a zweihander. There were no "greatswords" per se, it was just a generalization for D&D to classify big long heavy sword that requires two hands.
The bastard sword, or hand and a half sword, had a standard grip for one hand, but also tapered at the bottom with a hefty pommel that could be gripped by the second hand. If you look, once again at the Player's Handbook, that is the photo used by the longsword.
The longsword, historically, was not a one handed blade. The handle grip was designed for two hands. It was a lot lighter and more mobile than the two-handed sword.
When it comes to one handed swords, once again it is a generalization. The Anglo-Saxon Seax wasn't primarily a stabbing (piercing weapon). It was an edged weapon (slashing) and it has been noted as such due to the lack of handguard. Handguards, not only helped protect the fingers from another attack, but also prevented your hand from sliding up the blade when stabbing.
Look at this "short sword" (seax) from Owen Bush. There is very little to stop your hand from sliding off the grip if a stab is made. That would make stabbing with it less likely. Could you stab? Sure, why not. Is it optimal? No.
Look at this "short sword" (gladius) from Cult of Athena. There is a very pronounced handguard to prevent your hand from sliding forward. Also look at the very prominent piercing tip. That would make slicing with it less likely. Could you slice? Sure, why not. Is it optimal? No.
Reading the short swords description in the Player's Handbook will reveal that it is often used as an off-hand weapon. Due to the piercing nature and the european influence of D&D, the short sword in the player's handbook is most likely modeled after a main gauche.
Look at this main gauche from True Swords. It is a little longer than the typical dagger, with a very very deliberate pointed tip. Main gauches, in historical fencing, were designed to parry and stab in conjunction with a rapier, foil, epee, etc.
The wakizashi, in Oriental Adventures, is listed as having the same characteristics as a masterwork short sword. However, wakizashis were made along with and with similar techniques as the katana. One doesn't need to go far to see the cutting tests with the katana.
Look at this wakizashi from Casiberia. The wakizashi has the slight bend to the blade, emphasizing a cutting edge, rather than a piercing tip. Can you pierce with a wakizashi? Of course. Will it slice better? Absolutely.
Use Houserules for alternatives.
Wanting a thrusting short sword? Style it as a gladius. Want a slashing shortsword? Style it as a seax. Your culture and region of play in-game will determine what weapons are available, in use, and how they are used.
Best Answer
RAW, no, but a DM might allow it
Rules as Written, the longsword does slashing damage, and there are no rules that suggest otherwise (i.e. no weapon property that allows a different damage type), so it cannot deal bludgeoning damage.
However, if it seems reasonable to your DM, they might allow it. If I were your DM, I might allow it if your character had already seen other party members dealing bludgeoning damage and generally doing better for it, then roleplaying a brief scene where your character's like "Hmm, I can smash bones easier than I can cut them, maybe I should try holding my sword like this?", but ultimately whether this is allowed depends on each individual DM (some may allow it without any roleplay to justify it in-game, some may disallow it regardless because that's not RAW).
That said, if I were to allow it (and therefore if other DMs were to allow it, they may well rule the same), I would do as Erik's answer suggests; since, although one can use a sword in this way, it is not designed to be used in this way, so it should use the improvised weapon rules, which means it would do 1d4 bludgeoning damage instead of 1d8 slashing damage.