When boarding, I tend to aim for the Oxygen subsystem room - it's typically small, and critical enough that I will quickly attract attention. If they don't respond, then I've pretty much sealed their death quickly.
Meanwhile, I'll target whatever critical system I choose (usually weapons, shields, drones, or engines) with my laser weaponry. Once I've hurt one or more of their crewmembers, I'll target the medbay with missiles or bombs. That way, I can finish off heavily wounded crewmembers while disabling their ability to heal. Fire bombs and subsystem bombs are the best for this sort of attack.
I do believe that firing on the rooms where your boarders are will hurt them as well as the enemy, so I tend to avoid this. Trying to get the timing right for lasers (to reduce shields) + beams (to do damage) is also kind of tricky if you're also monitoring a boarding party.
Also, it helps if you've got a couple of teams of boarders - while you're healing the first squad, send the second squad in to keep the pressure up.
Some ships are particularly bad for trying to board, however. If they've got significant crew, especially a lot of mantises, I may give up trying to win by wiping them out. On the other hand, getting into an engagement and realizing they have no medbay pretty much means I've got a win locked down.
As you've noticed, Rock and Mantis crewmembers are the best to send as boarders. Don't forget that you can pause the game with the space bar, so if you think getting your crew back on board is going to be a close call, pausing and unpausing to slow the game down may help you slice the timing razor-thin.
Possible spoilers if you've never been to sector 8:
It's also been noted in a couple of strategies for fighting the final boss that the weapon systems are in isolated rooms with just a single human guarding each. If you send 2 of almost any race of boarders into these rooms, you can kill the guard and disable the system, thereby making this fight easier. The best weapon to start with is (in my opinion) the missile launcher, which is third from the left.
Breach bombs damage systems and crew, and cause cracks, but don't hurt hulls. Alternate bombing the medical bay and the O2 generator. Put priority on whichever one the crew is trying to repair. Eventually they'll have so little health they won't try to repair things, and you can just wait. If the ship doesn't have a medical bay it's even easier - just bomb the O2 generator until they stop trying to repair it.
If you're taking this tack, it helps to upgrade your own shields (and maybe have a defense drone) so you can sit and wait them out without having to waste missiles bombing their weapons as well.
Ion weapons can help, but it's hard to get enough of them to punch through 3-4 shields reliably, while you are still getting shot at. If you have them, great, but don't prioritize them.
You are correct about sensors. Don't even bother trying to kill crew without damaging the ship until you have the first sensor upgrade or a slug crew member.
Best Answer
The point of a Roguelike is that there isn't necessarily a winning strategy, you have to come up with it on the fly. Let me lay out the myriad ways one can die, and precautions against them:
Weapons
"Projectile" laser weapons. Your bread and butter. Every ship should have at least one, if not more. These are prime shield-breakers, just on shot-count alone. No one strategy is going to be perfect against them. Shields will soak them up... ablatively. You will dodge some of them. A Defense II drone can shoot them down somehow. Your best bet, though, is to knock out his weapons room. Missiles, shields, and engines will serve you well, and having certain drones might protect you in passing.
Beam Weapons. These are great capitalizing weapons. Shields down? Hope you don't like 8 armor and some subsystems! The only direct counter to these weapons is shields, which they can only penetrate to some extent. Fortunately, the ones with scary shield penetration values tend to not show up in the first half of the game, anyway.
Missiles. A great opening weapon, softening shield generators and gun rooms for barrages based on more renewable resources. The only counters to this weapon are evade and defense drones. Good engines give you as much as 50% evade, and drones are 150 for the controller plus varying amounts for the kind of drone you're interested in. Drones are preferable, but engines tend to be easier and cheaper.
Bombs. You can't stop them, you can only hope they miss. High evade values, especially with Cloak, are your only defense besides swiss cheesing his weapons room, which you should do with great prejudice. Missiles are the the weapon of choice for these guys, and you should always keep a couple on hand, in case.
Hazards
Teleporters. Not really a weapon, but they can mess your day up. Good airlock management, med-bay usage, and a spare crewman or two is the key here. If you can draw your enemies into the med-bay, they're basically done for. Have that upgraded, but only when you have spare change.
Fires. Have door upgrades researched, it's a pretty cheap buy. You should definitely have this by the end of the second zone.
Asteroids. Shields and evade. Mainly shields.
Asphyxiation. O2 controls.
As you can clearly see, there are a couple of consistencies:
Shields keep you alive
Engines keep you alive
Murdering the other ship keeps you alive
There are a few strategies you can try that seem to work for me:
Early investment in engines - anything that doesn't hit you is a return on investment. Follow that with investment in shields for maximum safety.
Cheaper, more-shot lasers are often more cost-effectiveness.
Invest a bit in everything, and a lot in a few things. This is called being "T shaped", and is considered a good standard in real life, too!
Teleporters are good if you can micromanage. Fortunately, you can pause the game whenever! Watch out for the automated ships, they're airless.
Sensors are more valuable than you'd think. If you've got spare change, consider them strongly.
Try and help the NPCs. Not every time you do will pay off, but it can pay off in a BIG way. Consider: Drone Recovery augment in the first stage with the Engi drone ship due to my going out of my way to help someone. It. Was. Glorious.
The message I hope you take away from this is that your question is, in a way, not the best one to ask. If this game was made well, there shouldn't be a real answer to your question. As it is, the best I can say is that some things have tended to work better for me, but at other times they've still failed spectacularly.