When boarding, I tend to aim for the Oxygen subsystem room - it's typically small, and critical enough that I will quickly attract attention. If they don't respond, then I've pretty much sealed their death quickly.
Meanwhile, I'll target whatever critical system I choose (usually weapons, shields, drones, or engines) with my laser weaponry. Once I've hurt one or more of their crewmembers, I'll target the medbay with missiles or bombs. That way, I can finish off heavily wounded crewmembers while disabling their ability to heal. Fire bombs and subsystem bombs are the best for this sort of attack.
I do believe that firing on the rooms where your boarders are will hurt them as well as the enemy, so I tend to avoid this. Trying to get the timing right for lasers (to reduce shields) + beams (to do damage) is also kind of tricky if you're also monitoring a boarding party.
Also, it helps if you've got a couple of teams of boarders - while you're healing the first squad, send the second squad in to keep the pressure up.
Some ships are particularly bad for trying to board, however. If they've got significant crew, especially a lot of mantises, I may give up trying to win by wiping them out. On the other hand, getting into an engagement and realizing they have no medbay pretty much means I've got a win locked down.
As you've noticed, Rock and Mantis crewmembers are the best to send as boarders. Don't forget that you can pause the game with the space bar, so if you think getting your crew back on board is going to be a close call, pausing and unpausing to slow the game down may help you slice the timing razor-thin.
Possible spoilers if you've never been to sector 8:
It's also been noted in a couple of strategies for fighting the final boss that the weapon systems are in isolated rooms with just a single human guarding each. If you send 2 of almost any race of boarders into these rooms, you can kill the guard and disable the system, thereby making this fight easier. The best weapon to start with is (in my opinion) the missile launcher, which is third from the left.
There is an augmentation "Weapon Pre-Igniter" that powers your weapons instantly when you come out of the jump. But even with this it is very hard to stop the teleporter. You need to either have sufficient firepower to bring down the shield and teleport station in one go. Or have enough shield bypassing or enough EMP firepower to be able to bring down the teleport station.
You also need to keep it down the whole battle, if it is repaired for a short time it will probably instantly teleport boarders on board. (This I think, I have not tested this).
When I had a ship with the Weapon Pre-Igniter, I never was able to stop the teleporter quick enough to prevent boarding from taking place. I have not tested this very extensively btw.
Taking out the teleport pad does prevent them from returning to their own ship, a blessing and a curse. (They do not teleport away before you can get the EXP. But on the other hand, you still have boarders on your ship)
The enemy can't teleport to a cloaked ship or retreat their troops from a cloaked ship. So you can use Stealth Weapons augmentation and cloaking to blast away the teleport pads of the enemy ship.
And, the Zoltan Shield also prevents ships from teleporting boarders on board. (Events still work).
Best Answer
Killing the crew in first phase means they won't be there in the next two, same if you kill them in second phase: they won't be there in the final, so killing them can be a good tactic as they may trouble you later.
If you kill them all you won't win that phase or overall, instead the ship will go into auto-pilot and will slowly repair all its systems much like the auto-assault scout drones in earlier sectors. It's best to leave one enemy crew alive, maybe the isolated guy on their laser tri-shot; because fighting an AI Boss is much harder than fighting a single panicky engineer.
When the ship flees to its next phase your boarding crew will be killed if they are still on board, same if you destroy it completely with your crew on board, meaning their noble sacrifice won't go down in the credits.