Why can’t I dominate certain law enforcement

The rules for this were changed 5th of March 2014 (emphasis mine):

Every player with the Dominator skill can now dominate one guard each at the same time. But apparently the old pager operators got fired, and more experienced operators are taking over. So clear your throats, and get ready to answer the pager as soon as you dominate a guard.

So one dominated guard or police officer per player with dominator. That means no, you cannot dominate multiple guards at once. You will need a friend to help.

Before this change the rule (which console and unpatched versions might still follow) was one dominated guard or police at a time. Want another one? Kill the existing one.

It's a % chance but you can have multiple attempts at dominating an enemy. The following factors seem to come into play when trying to dominate somebody:

• The level and type of enemy
• The number of enemies around at the time (large groups seem to just ignore you, while a guard by themselves will back down easier)
• The guard/police officers alert state (they're more likely to surrender if you jump out from around a corner at them and start screaming at them than if they're already actively engaged with you, but it's still possible to injure one and then shout him down)
• How far away you are from the guard/police officer
• The skills you have, the tier 3 bonus increases shout range and the tier 6 bonus increases intimidation amount

If you get too close to a guard, he will simply handcuff you and if you're too far away though (and don't have the top tier mastermind perk which increases intimidation and the additional skills that increase range) then you will simply be out of range.

Once you have your target's attention, intimidating them requires three shouts.

• The first shout will make him put his hands up and drop his weapon
• The second will make him kneel
• The third will make him handcuff himself

Once your target is handcuffed, he is a hostage and is included in the hostage counter, can be traded for your team mates lives, and can only be freed by a law enforcer. A handcuffed law enforcer can then be turned using the "Joker" ability in the Mastermind skill tree to fight on your side.