It gets factored in as part of the calculations only for the offhand if you don't have the Two Weapon Fighting feat, but honestly you should just ignore that wording as it's confusing. Two-Weapon Fighting spells it out much more clearly:
If you wield a second weapon in your off hand, you can get one extra
attack per round with that weapon. You suffer a -6 penalty with your
regular attack or attacks with your primary hand and a -10 penalty to
the attack with your off hand when you fight this way. You can reduce
these penalties in two ways:
If your off-hand weapon is light, the penalties are reduced by 2 each. (An unarmed strike is always considered light.)
The Two-Weapon Fighting feat lessens the primary hand penalty by 2, and the off-hand penalty by 6.
If you look carefully, the extra -4 is showing up in the first part, where your main hand gets -6 and your off-hand gets -10. It's still there with a light weapon (-4, -8). It disappears if you take the two weapon fighting feat, which reduces the main hand penalty by 2 and the off-hand by 6 (the difference is the -4 you noted being removed).
So while everything is being applied correctly, the wording of the part you noted is bad. Just use the Two-Weapon fighting entry itself, as it's wording is clear and the table summarizes very well what's going on.
The total penalties go like this, if you're curious:
Main-Hand: -2 for using two weapons, -2 for non light weapon, -2 for not having Two Weapon Fighting Feat = -6
Off-Hand: -2 for using two weapons, -2 for non light weapon, -2 for not having Two Weapon Fighting Feat, -4 for off-hand penalty (the rule you were curious about) = -10
edit - Using the example from the RotG article, you could argue (as the author does) that the off-hand penalty applies if you're wielding two weapons but only attacking with one of them (whichever weapon you picked as your off one). The off-hand would take the -4 for being offhand, but not the two weapon fighting penalties (as you're only attacking with one weapon). That's the cited example.
It never applies if you're only using one weapon, no matter what hand it's in, because "off-hand" is something that only appears under the two weapon fighting rules. The rules don't care which hand you use if you're only wielding one weapon.
By my reading, it also doesn't apply even if you're wielding two weapons but only using one of them to attack. RAW treats that as your main hand, no matter what. But if you wanted to use the Rules of the Game article's interpretation, then you'd have to pick an off hand as soon as you pick up a second weapon. Either way, when only wielding a single weapon it doesn't matter which hand you use.
In short, you are right and misinformed. The general rule is, when the situation fits the rule, and there is nothing in another rule to counter that, then you interpret the situation as in that rule.
In your case, when your off-hand weapon is light, -2/-2 is applied with TWF, and -4/-8 without it. Nowhere in the rule calls for the kind of weapon in the main hand, so you apply -2/-2 or -4/-8 (without the feat) no matter you are wielding a short sword or a longsword in the main hand.
Best Answer
Always as primary. Despite the fact that you are wielding another off-hand weapon, you are treated as one-handed, shield-less combat, as long as you declare it.
The loss in attack bonus is not given because of your 'bad' arm, but rather because you are constantly focusing both weapons, both arms, and the motion is very harder. This is why your readings of enemies are slower, and so you miss more. You are also more worried about disarming and etc etc.
Just remember to state beforehand in your turn that your next attack will be considered a one handed attack. If you do so, understand that until your next turn you are one handed. This also means you lose any other bonus inherited by dual wielding skills, such AC or reflex from some feats.
Also: As long as all your off hands match weigh limitations, any hand can become your primary. Easy with dagger/dagger because your offhand is always a light weapon. Since a mace is not a light weapon, it MUST be your primary weapon whenever dual wielding, unless you have proper feats to wield non-light weapons in your offhand without huge penalties.