When boarding, I tend to aim for the Oxygen subsystem room - it's typically small, and critical enough that I will quickly attract attention. If they don't respond, then I've pretty much sealed their death quickly.
Meanwhile, I'll target whatever critical system I choose (usually weapons, shields, drones, or engines) with my laser weaponry. Once I've hurt one or more of their crewmembers, I'll target the medbay with missiles or bombs. That way, I can finish off heavily wounded crewmembers while disabling their ability to heal. Fire bombs and subsystem bombs are the best for this sort of attack.
I do believe that firing on the rooms where your boarders are will hurt them as well as the enemy, so I tend to avoid this. Trying to get the timing right for lasers (to reduce shields) + beams (to do damage) is also kind of tricky if you're also monitoring a boarding party.
Also, it helps if you've got a couple of teams of boarders - while you're healing the first squad, send the second squad in to keep the pressure up.
Some ships are particularly bad for trying to board, however. If they've got significant crew, especially a lot of mantises, I may give up trying to win by wiping them out. On the other hand, getting into an engagement and realizing they have no medbay pretty much means I've got a win locked down.
As you've noticed, Rock and Mantis crewmembers are the best to send as boarders. Don't forget that you can pause the game with the space bar, so if you think getting your crew back on board is going to be a close call, pausing and unpausing to slow the game down may help you slice the timing razor-thin.
Possible spoilers if you've never been to sector 8:
It's also been noted in a couple of strategies for fighting the final boss that the weapon systems are in isolated rooms with just a single human guarding each. If you send 2 of almost any race of boarders into these rooms, you can kill the guard and disable the system, thereby making this fight easier. The best weapon to start with is (in my opinion) the missile launcher, which is third from the left.
Breach bombs damage systems and crew, and cause cracks, but don't hurt hulls. Alternate bombing the medical bay and the O2 generator. Put priority on whichever one the crew is trying to repair. Eventually they'll have so little health they won't try to repair things, and you can just wait. If the ship doesn't have a medical bay it's even easier - just bomb the O2 generator until they stop trying to repair it.
If you're taking this tack, it helps to upgrade your own shields (and maybe have a defense drone) so you can sit and wait them out without having to waste missiles bombing their weapons as well.
Ion weapons can help, but it's hard to get enough of them to punch through 3-4 shields reliably, while you are still getting shot at. If you have them, great, but don't prioritize them.
You are correct about sensors. Don't even bother trying to kill crew without damaging the ship until you have the first sensor upgrade or a slug crew member.
Best Answer
They stay in the enemy ship, but they stop attacking. You can move them around on the enemy ship, and (if you have sufficient sensors) you will still see all rooms inside of the opponents ship. Once you teleport them back, the internal ship view ends and it looks like a normal concluded battle.
It is amusing to have Lanius boarders: you can accept a surrender, and just leave your boarders in with the enemy characters until they asphyxiate. However this does not trigger any event or new rewards. The game considers the encounter over once you accept the surrender.