Your examples are distinct
Damage rolls, in particular, are completely different from any other check, since they do not use a d20, but rather anywhere from 1d2 to 2d6 (and that’s just for player-race-sized options!) plus various “damage bonuses” that vary from weapon to weapon (non-composite projectile weapons get none, light weapons get half-Strength, one-handed weapons get Strength, two-handed weapons get half-again-Strength, composite projectile weapons get a fixed number, and exceptions and special cases exist for each) and from class to class (rogues might add Sneak Attack damage, rangers may add Favored Enemy bonuses, etc).
But attack rolls are also distinct from any ability checks, whether that be Strength or Dexterity. For one thing, they use Base Attack Bonus and auto-succeed on a nat-20 and auto-fail on a nat-1, and have critical threats and critical hits, and all kinds of other things that apply “attack bonuses,” and ability checks do none of these things.
Skills too, for all they represent almost the same thing as ability checks, just with training added, are not technically ability checks. Bonuses to ability checks wouldn’t apply to skill checks (which is why almost every single bonus to ability checks is actually explicitly a bonus to ability checks and to skill checks using that ability).
Some rolls are subsets or specific types of other types
The big one is Initiative, which is a Dexterity ability check. There probably are other examples. But in every case, this is explicitly noted:
Initiative Checks
At the start of a battle, each combatant makes an initiative check. An initiative check is a Dexterity check.
(emphasis and emphasis mine)
As I'm wielding a crossbow, would I add my attack missile bonus (+12)
to my attack roll to see if I get past the AC, or to the damage roll
to see how much damage I deal?
The listed missile attack bonus is used to determine if you hit something; it's added only to your attack rolls. The damage rolls are determined seperately. For a crossbow, by default the damage is only the flat roll from the weapon.
You add the magical enhancement bonus, if it has one, to both the attack roll and the damage.
It's even weirder for bows. When shooting with a bow, you also attack with the listed missile attack bonus, but you have to use your strength modifier on the damage roll if it is negative. Which means with a bow your character would deal -1 damage because of low strength, but a high strength character would not add his strength to the damage roll unless it's a special (expensive) Composite Bow.
Hope this helps clear it up a little.
Best Answer
Grappling and shoving are the only RAW ways to "attack with" a skill
The Grappling rules state:
The rules on shoving a creature immediately follow the grappling rules:
Using grappling and shoving, you can make use of your Athletics skill in combat to prevent an enemy from moving and/or knock it prone to grant advantage to attacks from within 5 feet. However, there are no other ways, by RAW, to "attack with" a skill (whether Athletics or any other skill); all other attacks involve an attack roll instead of an ability check.