This is mostly based on a lot of the lore present in 3.5. I believe it was designed to incorporate mostly Forgotten Realms (as many of the canon gods and people are in both), but I could be wrong.
But this rule is probably based on Forgotten Realms. In that setting, the hand crossbow is a weapon designed and used almost exclusively by the Drow (dark elves). Therefore they may have decided to keep that idea. Look at the entry in the Monster Manual for Drow and you'll see they use the hand crossbow.
It may also be a matter of how it fires. The recoil from that might be hard to control with only one hand.
(I also can't help but think of the Cleric Quintet by R.A. Salvatore, in which someone builds a hand crossbow based on a picture of one, and there are times when people are actually surprised to see a human using one instead of a Drow. The Legend of Drizz't also makes many references to it being a popular tactic of the Drow to fire a single poisoned shot with it before engaging in melee).
Everybody needs a martial weapon proficiency
The shield bash rules say this (emphasis mine):
You can bash an opponent with a light shield or heavy shield, using it
as an off-hand weapon. See Table: Weapons for the damage dealt by a
shield bash. Used this way, a shield is a martial bludgeoning weapon.
For the purpose of penalties on attack rolls, treat a heavy shield as
a one-handed weapon and a light shield as a light weapon. If you use
your shield as a weapon, you lose its AC bonus until your next action
(usually until the next round). An enhancement bonus on a shield does
not improve the effectiveness of a shield bash made with it, but the
shield can be made into a magic weapon in its own right
Shield proficiency says this:
Benefit You can use a shield and take only the standard penalties.
Normal When you are using a shield with which you are not proficient, you take the shield’s armor check penalty on attack rolls
and on all skill checks that involve moving, including Ride checks.
Shield proficiency doesn't give you any kind of weapon proficiency with shields. It gives you the ability to use the shield and not take the armor check penalty. In order to not take a penalty for using a shield bash, you need that martial weapon proficiency. Classes that give you all martial weapons take care of that pretty easily, like a level of Fighter (or Radiant Servant of Pelor if you're a Cleric).
What about "Simple Weapons Only"?
The other classes you mentioned actually have a list of what they're proficient in, the Cleric's "simple weapons only" is just shorthand for a list that has every single simple weapon on it. It doesn't mean that Shield Proficiency works differently for Clerics than it does for everybody else.
It's also worth noting that the Cleric says "all simple weapons", not "only simple weapons". It's not a special rule or exclusion, it's just a shorter way of saying it than listing every simple weapon.
Pathfinder - The Same
From looking around, this appears to be the same in Pathfinder. The shield bash rules are basically the same, the shield proficiency rules are worded a bit differently but don't affect this, and James Jacobs (Creative Director at Paizo) agrees.
In fact, the Paizo FAQ states clearly that shield proficiency != weapon proficiency with shields.
Best Answer
Both light and hand crossbows take less time to reload than a heavy crossbow does.
As far as the tactical advantage of a hand crossbow, you can use it with one hand without a penalty, which allows you to theoretically take advantage of it in applications that would otherwise be difficult. For instance, a light crossbow takes a -2 penalty to fire one-handed, and a heavy crossbow takes a -4 penalty.
If you were to, say, be climbing up a wall or the like, you could use the hand crossbow with just one hand and then fire it without a penalty, though you'd still need to free the other hand to reload.
Combined with sneak attacks, which are how rogues can do a lot of damage, you can actually make good use of a hand crossbow to attack from unexpected, hard to retaliate against positions.
Hand crossbows can also be hidden with Sleight of Hand, and are explicitly mentioned in the text; light crossbows are not technically light weapons (except for firing one with each hand) and so cannot be hidden in this way.
That said, there's not a whole lot of applications where hand crossbows excel, but since 3.5 isn't a game where weapon type is going to be your main source of damage by end-game, sacrificing some damage on the die is not necessarily going to be as painful as one would think, especially since hand crossbows can still carry the same enchantments and other bonuses as other crossbows. The only other sacrifice you make for a hand crossbow is range, which won't typically be an issue because you want to be in that 30 foot sneak attack distance.