[RPG] How to calculate attack roll for a monster

You are probably looking at a ray spell. The Magic rules state:

Ray

Some effects are rays. You aim a ray as if using a ranged weapon, though typically you make a ranged touch attack rather than a normal ranged attack. As with a ranged weapon, you can fire into the dark or at an invisible creature and hope you hit something. You don't have to see the creature you're trying to hit, as you do with a targeted spell. Intervening creatures and obstacles, however, can block your line of sight or provide cover for the creature at which you're aiming.

If a ray spell has a duration, it's the duration of the effect that the ray causes, not the length of time the ray itself persists.

If a ray spell deals damage, you can score a critical hit just as if it were a weapon. A ray spell threatens a critical hit on a natural roll of 20 and deals double damage on a successful critical hit.

Other ranged spells will state whether it's a ranged attack or a ranged touch attack, which determines whether you are trying to hit the enemy's normal AC or their touch AC. In any event, it is a ranged attack and therefore the Combat rules state:

Your attack bonus with a ranged weapon is:

Base attack bonus + Dexterity modifier + size modifier + range penalty

Unless you have a special ability that says otherwise, you do not add anything to damage. See Damage in the Combat rules - it says you add Strength bonus to melee or thrown weapons, period.

Though you should add more context to your question - just in case you're super confused, let me clarify that you don't need to roll to hit on any spell just because it has a range. Take Fireball, range Long. You don't roll to hit with it, you just place the 20' radius spread area wherever you want within that range (that you have line of effect to yadda yadda) and it affects everyone in that area.

As I'm wielding a crossbow, would I add my attack missile bonus (+12) to my attack roll to see if I get past the AC, or to the damage roll to see how much damage I deal?

The listed missile attack bonus is used to determine if you hit something; it's added only to your attack rolls. The damage rolls are determined seperately. For a crossbow, by default the damage is only the flat roll from the weapon.

You add the magical enhancement bonus, if it has one, to both the attack roll and the damage.

It's even weirder for bows. When shooting with a bow, you also attack with the listed missile attack bonus, but you have to use your strength modifier on the damage roll if it is negative. Which means with a bow your character would deal -1 damage because of low strength, but a high strength character would not add his strength to the damage roll unless it's a special (expensive) Composite Bow.

Hope this helps clear it up a little.