[RPG] How are damage penalties on melee attacks calculated

It gets factored in as part of the calculations only for the offhand if you don't have the Two Weapon Fighting feat, but honestly you should just ignore that wording as it's confusing. Two-Weapon Fighting spells it out much more clearly:

If you wield a second weapon in your off hand, you can get one extra attack per round with that weapon. You suffer a -6 penalty with your regular attack or attacks with your primary hand and a -10 penalty to the attack with your off hand when you fight this way. You can reduce these penalties in two ways:

If your off-hand weapon is light, the penalties are reduced by 2 each. (An unarmed strike is always considered light.) The Two-Weapon Fighting feat lessens the primary hand penalty by 2, and the off-hand penalty by 6.

If you look carefully, the extra -4 is showing up in the first part, where your main hand gets -6 and your off-hand gets -10. It's still there with a light weapon (-4, -8). It disappears if you take the two weapon fighting feat, which reduces the main hand penalty by 2 and the off-hand by 6 (the difference is the -4 you noted being removed).

So while everything is being applied correctly, the wording of the part you noted is bad. Just use the Two-Weapon fighting entry itself, as it's wording is clear and the table summarizes very well what's going on.

The total penalties go like this, if you're curious:

Main-Hand: -2 for using two weapons, -2 for non light weapon, -2 for not having Two Weapon Fighting Feat = -6

Off-Hand: -2 for using two weapons, -2 for non light weapon, -2 for not having Two Weapon Fighting Feat, -4 for off-hand penalty (the rule you were curious about) = -10

edit - Using the example from the RotG article, you could argue (as the author does) that the off-hand penalty applies if you're wielding two weapons but only attacking with one of them (whichever weapon you picked as your off one). The off-hand would take the -4 for being offhand, but not the two weapon fighting penalties (as you're only attacking with one weapon). That's the cited example.

It never applies if you're only using one weapon, no matter what hand it's in, because "off-hand" is something that only appears under the two weapon fighting rules. The rules don't care which hand you use if you're only wielding one weapon.

By my reading, it also doesn't apply even if you're wielding two weapons but only using one of them to attack. RAW treats that as your main hand, no matter what. But if you wanted to use the Rules of the Game article's interpretation, then you'd have to pick an off hand as soon as you pick up a second weapon. Either way, when only wielding a single weapon it doesn't matter which hand you use.

Unless something specifically says otherwise, fractions are always rounded down. See The Basics:

Rounding Fractions

In general, if you wind up with a fraction, round down, even if the fraction is one-half or larger.

Exception: Certain rolls, such as damage and hit points, have a minimum of 1.

So if you have Str 16, a greatsword will add +4 bonus to damage (rounded down from 4½).

Note that this round down, not towards zero. This matters for negative numbers.

For example, negative Strength mods don't get multiplied because the rule says not to, but if they did and you have Str 8, a dagger would add a −1 penalty (rounded down from −½).