Definitions
We all have our limits and boundaries. Lines and veils are different ways to handle those boundaries in play.
A line is, well, a line — a hard limit, something we do not want to cross. Lines represent places we don't want to go in roleplaying.
"There is no torture in the events in our game. We don't do it, NPCs don't do it to us or to each other. Whether it happens elsewhere in the setting is not an issue in terms of enjoying play."1
A veil is a "pan away" or "fade to black" moment. When we veil something, we're making it a part of the story, but keeping it out of the spotlight. Think of it as a way to still deal with certain themes while avoiding having to describe them in graphic detail.
"Torture does happen in the game world and may happen in our game in some way or another. But if and when it does, we do not role-play it directly or depict it verbally. Everyone is trusted to play their characters as reacting to it appropriately without us having to experience it vicariously."1
Background
This terminology came out of Forge discussion some years ago, plus Sex and Sorcery, a supplement to the game Sorcerer. It's a feature of indie-RPG discussion because the community strives to be inclusive but also features a lot of games that deal with difficult content.
Techniques in Play
You can establish some limits ahead of time, as you can see in the examples above. This is a good approach for stuff you know will come up, because it's central to the game system, the setting, or the genre you have in mind. Oftentimes groups do this using shorthand like "Let's keep it PG-13."
Where an understanding of lines and veils really shines, though, is in helping you to communicate about issues as they arise during play. This is something every gamer should learn since it's part of looking out for each other's fun, and there's no way to perfectly plan your way around every issue before the game starts.
The most important thing is to make it easy to speak up. That means you should cultivate a non-judgmental atmosphere. A basic lines-and-veils discussion kinda looks like this:
"Ouch!"
"Oops. Sorry. Let's fix that."
Don't try to argue someone to give in on their limits. Don't try to assign blame to anyone. The person who's feeling uncomfortable may want to talk about it, or they might not; either way is okay. Identify the problem, fix the problem in a way that addresses their specific needs and desires, make a mental note of it for next time, and continue play.
In many cases, a group can do fine without formal structure for this. However, for tricky subjects or gaming with strangers, formal systems are a big help. Variations include:
Really, lines and veils in play are about communication, attitude, and interpersonal skills:
The Veil or The X-Card doesn't replace our complicated interpersonal toolsets that we've spent our whole lives developing... it signals that now is the time to use them. Maybe the stock solution of "edit out the problem content, keep moving" works, but maybe not. It's case + person + issue + game-dependent.2
However you communicate about limits and comfort, remember that signaling devices aren't a perfect substitute for social awareness. Just because someone isn't explicitly telling you they have a problem doesn't mean everything is fine always. A shy player or someone who's been surprised by a trauma trigger might not be able to assert a line in the moment. So keep an eye on people's actual reactions and adjust your play accordingly.
1 - Examples from Sorcerer author Ron Edwards.
2 - Commentary from Monsterhearts author Avery Alder. Check out the free mini-supplement Safe Hearts (PDF link) for more advice about dealing with emotionally challenging content.
Best Answer
Origins
The original "bag of rats" trick was a thought experiment in third edition Dungeons and Dragons involving the feats Great Cleave and Whirlwind Attack, which proceeded about as follows:
The "bag of rats" was how you surrounded yourself in this way - carrying around a bag of untamed rats and dumping it out.
Principles
This takes advantage of an assumption made by the designers: that killing blows would be comparatively rare events, and therefore any mechanical benefits from striking a killing blow could be given an absolute value. In the case of Great Cleave, the benefit was a follow-up attack.
In the general case, a "bag of rats" is a collection of notionally hostile but essentially helpless targets, used to safely acquire any mechanical benefits gained from hitting or killing an enemy at a trivial additional risk.
Adjudication
Fourth edition D&D, with its many tactical powers and benefits on hit, ultimately just advised the DM that they shouldn't let powers grant these benefits if the PCs are trying to hit something that the DM doesn't consider a threat. That's ultimately where this has to be stopped - an arbitrary DM call with no mechanical absolutes.
You can try to intercept this at a higher level - like "that's a swarm of rats, effectively a single creature, whirlwind hits it once" - but that still leaves the door open. The benefits are there for hitting and killing opposition the DM put there, not for arbitrary player-declared targets.