[RPG] When adding 1-1/2 or 1/2 your strength mod to damage how does it round

No, for three reasons:

First, Rapier specifically states:

You can’t wield a rapier in two hands in order to apply 1½ times your Strength bonus to damage.

Second, Weapon Finesse allows you to use DEX mod instead of STR mod on attack rolls. Wielding a one handed weapon in two hands allows you to add to your damage rolls. Two completely different cases.

Third, you cannot use Weapon Finesse when wielding two handed weapons, or wielding a one handed weapon in two hands. There are a couple of specific items that change this ( iirc, Elven Courtblade from Races of the Wild and the cheesechain Spiked Chain ), but it is limited to those items.

The Strength Penalty to Damage Remains Unadjusted
A Medium creature with Strength 8 who's fighting with two weapons, a longsword in one hand and a short sword in his off-hand, will inflict 1d8-1 points of damage with the longsword and 1d6-1 points of damage with the short sword, for example. The same creature with Strength 6 will inflict 1d8-2 points of damage with the longsword and 1d6-2 points of damage with the short sword.

From the SRD.

Strength Bonus
When you hit with a melee or thrown weapon, including a sling, add your Strength modifier to the damage result.

Off-Hand Weapon
When you deal damage with a weapon in your off hand, you add only ½ your Strength bonus.

Wielding a Weapon Two-Handed
When you deal damage with a weapon that you are wielding two-handed, you add 1½ times your Strength bonus. However, you don’t get this higher Strength bonus when using a light weapon with two hands.

Also from the SRD:

Modifiers
A modifier is any bonus or penalty applying to a die roll. A positive modifier is a bonus, and a negative modifier is a penalty.

Emphasis--except in the headers--mine. Thus while the bonus may be different, the other kind of modifier--the penalty--is unchanged.