When boarding, I tend to aim for the Oxygen subsystem room - it's typically small, and critical enough that I will quickly attract attention. If they don't respond, then I've pretty much sealed their death quickly.
Meanwhile, I'll target whatever critical system I choose (usually weapons, shields, drones, or engines) with my laser weaponry. Once I've hurt one or more of their crewmembers, I'll target the medbay with missiles or bombs. That way, I can finish off heavily wounded crewmembers while disabling their ability to heal. Fire bombs and subsystem bombs are the best for this sort of attack.
I do believe that firing on the rooms where your boarders are will hurt them as well as the enemy, so I tend to avoid this. Trying to get the timing right for lasers (to reduce shields) + beams (to do damage) is also kind of tricky if you're also monitoring a boarding party.
Also, it helps if you've got a couple of teams of boarders - while you're healing the first squad, send the second squad in to keep the pressure up.
Some ships are particularly bad for trying to board, however. If they've got significant crew, especially a lot of mantises, I may give up trying to win by wiping them out. On the other hand, getting into an engagement and realizing they have no medbay pretty much means I've got a win locked down.
As you've noticed, Rock and Mantis crewmembers are the best to send as boarders. Don't forget that you can pause the game with the space bar, so if you think getting your crew back on board is going to be a close call, pausing and unpausing to slow the game down may help you slice the timing razor-thin.
Possible spoilers if you've never been to sector 8:
It's also been noted in a couple of strategies for fighting the final boss that the weapon systems are in isolated rooms with just a single human guarding each. If you send 2 of almost any race of boarders into these rooms, you can kill the guard and disable the system, thereby making this fight easier. The best weapon to start with is (in my opinion) the missile launcher, which is third from the left.
Breach bombs damage systems and crew, and cause cracks, but don't hurt hulls. Alternate bombing the medical bay and the O2 generator. Put priority on whichever one the crew is trying to repair. Eventually they'll have so little health they won't try to repair things, and you can just wait. If the ship doesn't have a medical bay it's even easier - just bomb the O2 generator until they stop trying to repair it.
If you're taking this tack, it helps to upgrade your own shields (and maybe have a defense drone) so you can sit and wait them out without having to waste missiles bombing their weapons as well.
Ion weapons can help, but it's hard to get enough of them to punch through 3-4 shields reliably, while you are still getting shot at. If you have them, great, but don't prioritize them.
You are correct about sensors. Don't even bother trying to kill crew without damaging the ship until you have the first sensor upgrade or a slug crew member.
Best Answer
Shooters always fire at a random enemy. In 2 vs. 1, there's no difference, because there's only one enemy. In 3 vs. 2, or other situations, the shooter is supposed to distribute damage evenly by shooting randomly, but because of how random numbers work, one of the targets will end up with more damage than the other. (Random is not the same as evenly-distributed or nice.) This makes the other combat(s) in the room suddenly become much more lopsided after somebody dies.