Why is it “hats off” and not “hat’s off”

Compare:

Even people off to the sides of these stage speakers enjoy well-balanced, detailed sound...

to:

Even people to the sides of these stage speakers enjoy well-balanced, detailed sound...

In this case, there's the chance of misinterpretation if off is left out. It could sound like the people are directly at the sides of the stage speakers. Off to the sides, however, clarifies that there is distance at play and makes the sentence make more sense.

In other cases where it sounds unnecessary, it may be kept for sonorous purposes. In other words, we say it because it gives the feeling of the language flowing or maintains the rhythm of the sentence.

As another example, compare:

The attack came not from the monster before her, but from those off to the sides.

and

The attack came not from the monster before her, but from those to the sides.

In the first example, it's implying a long-range attack; are they shooting her with an arrow?

In the second example, it once again doesn't sound as far away. Maybe they're directly to either side of her or either side of the monster in front of her. In this case, it could conceivably be a close-range attack.

In the words of the Oxford English Dictionary,

After transitive verbs, the person or thing affected (‘secondary object’) is often introduced by of.

Of represents what was in earlier stages of the language a genitive. Other verbs which have this feature include cheat, defraud, accuse, suspect and avail.