It gets factored in as part of the calculations only for the offhand if you don't have the Two Weapon Fighting feat, but honestly you should just ignore that wording as it's confusing. Two-Weapon Fighting spells it out much more clearly:
If you wield a second weapon in your off hand, you can get one extra
attack per round with that weapon. You suffer a -6 penalty with your
regular attack or attacks with your primary hand and a -10 penalty to
the attack with your off hand when you fight this way. You can reduce
these penalties in two ways:
If your off-hand weapon is light, the penalties are reduced by 2 each. (An unarmed strike is always considered light.)
The Two-Weapon Fighting feat lessens the primary hand penalty by 2, and the off-hand penalty by 6.
If you look carefully, the extra -4 is showing up in the first part, where your main hand gets -6 and your off-hand gets -10. It's still there with a light weapon (-4, -8). It disappears if you take the two weapon fighting feat, which reduces the main hand penalty by 2 and the off-hand by 6 (the difference is the -4 you noted being removed).
So while everything is being applied correctly, the wording of the part you noted is bad. Just use the Two-Weapon fighting entry itself, as it's wording is clear and the table summarizes very well what's going on.
The total penalties go like this, if you're curious:
Main-Hand: -2 for using two weapons, -2 for non light weapon, -2 for not having Two Weapon Fighting Feat = -6
Off-Hand: -2 for using two weapons, -2 for non light weapon, -2 for not having Two Weapon Fighting Feat, -4 for off-hand penalty (the rule you were curious about) = -10
edit - Using the example from the RotG article, you could argue (as the author does) that the off-hand penalty applies if you're wielding two weapons but only attacking with one of them (whichever weapon you picked as your off one). The off-hand would take the -4 for being offhand, but not the two weapon fighting penalties (as you're only attacking with one weapon). That's the cited example.
It never applies if you're only using one weapon, no matter what hand it's in, because "off-hand" is something that only appears under the two weapon fighting rules. The rules don't care which hand you use if you're only wielding one weapon.
By my reading, it also doesn't apply even if you're wielding two weapons but only using one of them to attack. RAW treats that as your main hand, no matter what. But if you wanted to use the Rules of the Game article's interpretation, then you'd have to pick an off hand as soon as you pick up a second weapon. Either way, when only wielding a single weapon it doesn't matter which hand you use.
As I'm wielding a crossbow, would I add my attack missile bonus (+12)
to my attack roll to see if I get past the AC, or to the damage roll
to see how much damage I deal?
The listed missile attack bonus is used to determine if you hit something; it's added only to your attack rolls. The damage rolls are determined seperately. For a crossbow, by default the damage is only the flat roll from the weapon.
You add the magical enhancement bonus, if it has one, to both the attack roll and the damage.
It's even weirder for bows. When shooting with a bow, you also attack with the listed missile attack bonus, but you have to use your strength modifier on the damage roll if it is negative. Which means with a bow your character would deal -1 damage because of low strength, but a high strength character would not add his strength to the damage roll unless it's a special (expensive) Composite Bow.
Hope this helps clear it up a little.
Best Answer
Your examples are distinct
Damage rolls, in particular, are completely different from any other check, since they do not use a d20, but rather anywhere from 1d2 to 2d6 (and that’s just for player-race-sized options!) plus various “damage bonuses” that vary from weapon to weapon (non-composite projectile weapons get none, light weapons get half-Strength, one-handed weapons get Strength, two-handed weapons get half-again-Strength, composite projectile weapons get a fixed number, and exceptions and special cases exist for each) and from class to class (rogues might add Sneak Attack damage, rangers may add Favored Enemy bonuses, etc).
But attack rolls are also distinct from any ability checks, whether that be Strength or Dexterity. For one thing, they use Base Attack Bonus and auto-succeed on a nat-20 and auto-fail on a nat-1, and have critical threats and critical hits, and all kinds of other things that apply “attack bonuses,” and ability checks do none of these things.
Skills too, for all they represent almost the same thing as ability checks, just with training added, are not technically ability checks. Bonuses to ability checks wouldn’t apply to skill checks (which is why almost every single bonus to ability checks is actually explicitly a bonus to ability checks and to skill checks using that ability).
Some rolls are subsets or specific types of other types
The big one is Initiative, which is a Dexterity ability check. There probably are other examples. But in every case, this is explicitly noted:
(emphasis and emphasis mine)