A lot of this is dependent on class. As a Vanguard for example, you'll probably never want to use an Assault Rifle - they're just too heavy, and as a Soldier, you'll usually give SMG's a pass - everything they can do an AR does better. Meanwhile, an Infiltrator gets lots of bonuses when using their trusty Sniper Rifle, evening if irresponsibly shooting it from the hip.
The second consideration is range: at very long ranges, a Sniper Rifle offers top notch accuracy. AR's and Pistols Can be pretty good. SMG's can be flaky though, and shotguns are nigh useless. By contrast, Shotgubs ate devastating at short range, but the loss of peripheral vision while looking down that sniper scope can get you killed.
The next consideration is the defenses of your opponents. Slow firing, high caliber weapons like Sniper Rifles and Pistols chew through Armor, but are weak against Biotic Barriers and Shields. Rapid fire weapons with a lot of projectiles, by contrast, like SMG's and Shotguns chew up Shields and Barriers quickly, but have trouble with armor. Assault Rufles are a bit of an odd duck as they're 'pretty good' at handling everything.
Additionally there are mods availiable for several weapon types that alter their capabilities. You can add Armor-penetration to an SMG or a shotgun, or a melee upgrade to a pistol or shotgun. Maybe a scope on your SMG to improve its mid-long range performance?
Finally, you may want to pair some weapons up with certain specialty ammo. For instance, SMG's are devestating in combination with Cryo or Incendiary Ammo as their high rate of fire gives them more opportunities to proc the special effects provided by these ammo types than, say, a slow firing shotgun.
Any projectiles which benefit from the piercing (in this case, penetrate a .5 meter thick object) will deal reduced damage (60%). Projectiles which don't need to penetrate cover or an obstacle to hit their target do normal damage. This includes the visual armor that Husks / Cannibals can get, and the shields of Cerberus Guardians.
Additionally, depending on the type of enemy and the difficulty, some enemies have "Armor" (not to be confused with the yellow Armor bar some enemies have on top of / instead of health). This armor reduces damage by a flat amount, and it is this Armor that the piercing mod works against. So you do get some benefit when attacking all colors of health bar.
Best Answer
You can't carry Heavy Weapons around in this game. If you switch from them, you drop them.
When you pick up a Heavy Weapon, it temporarily replaces your current weapon from the weapon select wheel - the reason you couldn't find your Assault Rifle was because you technically still had it as your active weapon.
So you then switched to your pistol, which dropped the Heavy Weapon (on the ground nearby, able to be picked up again).
After this, your Assault Rifle should have been selectable again. Did you only try to switch to the assault rifle while you were carrying the Reaper Blackstar (the Heavy Weapon you found)?