How does the “Double damage in combat” trait work for the Mantis crew in FTL

The combat effectiveness of your crew varies, and additionally they level up as they fight.

For instance, Engi and Zoltan crew members have lower health than Humans, Slugs, Mantis, or Rock crew members. Mantis crew also have a bonus to combat damage. Engi and Zoltan especially are worthless in a fight - be ready to move them quickly, and plan on any combat taking a while if you can't get the enemies into the medbay quickly.

I tend to assign at least one, but ideally two of my crew members to "security detail" - they're the ones that get pulled into rooms to fight if there are enemy boarders. They're also typically my away team if I have a teleporter. I will tend to pick Mantis first for this, then Rock, and finally Human. If I've got to double up and have my subsystem crew work security detail, I tend to leave the weapon officer and pilot alone - other crewmates will pull security detail before I'll move them.

Some other tips:

• As has been noted, the best place to fight is in your medbay, as you'll constantly regenerate health so long as the medbay is undamaged and powered up.
• I will sometimes open doors to route enemies into 2-square rooms, or let them have a particular subsystem (ie, sensors or door control) if they're likely to move into a more advantageous room after they've damaged it.
• Remember that you can pause (with space) and issue orders - if you need to shuffle crew around, and you've got a lot going on between ship combat and dealing with boarders, this can save your crew, especially if you're like me and you tend to click doors rather than crew in a panic.
• In combat, you'll still take damage as long as you're in the room with the enemy, so make sure you leave enough health on your crew to make it out of the room before they die.
• If you've decompressed part of the ship, remember that you'll take damage passing through it, so keep that in mind as well if the decompressed portions of the ship are between your crew and the medbay.
• Upgrade doors will help a lot, giving you more time to "vent" them to death or time for you to finish the fight with the ship before taking care of the boarders. Just remember robots don't need oxygen and if your door controls get destroyed you can't close doors, which means you could lose oxygen.

When boarding, I tend to aim for the Oxygen subsystem room - it's typically small, and critical enough that I will quickly attract attention. If they don't respond, then I've pretty much sealed their death quickly.

Meanwhile, I'll target whatever critical system I choose (usually weapons, shields, drones, or engines) with my laser weaponry. Once I've hurt one or more of their crewmembers, I'll target the medbay with missiles or bombs. That way, I can finish off heavily wounded crewmembers while disabling their ability to heal. Fire bombs and subsystem bombs are the best for this sort of attack.

I do believe that firing on the rooms where your boarders are will hurt them as well as the enemy, so I tend to avoid this. Trying to get the timing right for lasers (to reduce shields) + beams (to do damage) is also kind of tricky if you're also monitoring a boarding party.

Also, it helps if you've got a couple of teams of boarders - while you're healing the first squad, send the second squad in to keep the pressure up.

Some ships are particularly bad for trying to board, however. If they've got significant crew, especially a lot of mantises, I may give up trying to win by wiping them out. On the other hand, getting into an engagement and realizing they have no medbay pretty much means I've got a win locked down.

As you've noticed, Rock and Mantis crewmembers are the best to send as boarders. Don't forget that you can pause the game with the space bar, so if you think getting your crew back on board is going to be a close call, pausing and unpausing to slow the game down may help you slice the timing razor-thin.

Possible spoilers if you've never been to sector 8:

It's also been noted in a couple of strategies for fighting the final boss that the weapon systems are in isolated rooms with just a single human guarding each. If you send 2 of almost any race of boarders into these rooms, you can kill the guard and disable the system, thereby making this fight easier. The best weapon to start with is (in my opinion) the missile launcher, which is third from the left.