The composition of your team is the first line of defense against Anti-Mage. Having a good core of gank-tasked heroes (3+) and ideally, no more than 1 hard carry. With Anti-Mage, like many of the late-game carries (Spectre, Void, Antimage), your game should not fixate on countering him late game, but instead ask what counters Anti-Mage?

Wards, Ganks, and Pressure

You need to shut down these hard late-game carries early. Even allowing the completion of a single core item uninterrupted is a bad sign. Where you listed 3 high tiered items, the momentum for Anti-Mage was likely far out of your control to rein in late game.

So you're looking to gank him frequently. Note that if the enemy team has more than one late-game carry this becomes more difficult for both teams, where your team will need to keep both late-game heroes in check and from the enemy team's side they need to [1] not die, and [2] wait you out. So logically, you need to do the opposite, gank them far more often than you would consider necessary and push down towers.

While "kill them and push towers" is a bit too simple-minded of an approach, the crux of the issue is defined.

The problem is that in execution, as is commonly the case especially on randomly assembled teams, not many players risk leaving their lane, leaving behind their potential farm-able gold, and losing experience roaming from one lane to another.

But what is important here is to reframe the issue, again, and that is a successful gank does not always need to result in a kill. Pushing Anti-Mage out of the lane to hug the tower, deny his gold/experience gain, return to base to heal, or even just added pressure that you guys are coming for him stacks up for an effect in your favor. Understanding that 'yes he has blink' and 'no you probably won't kill him' you are primarily focused on disrupting his acquisition of gold and experience.

Laning against Anti-Mage should involve 2 heroes that can stun or disable, understand when picking a lineup against Void, Anti-Mage, or even Spectre, their escape mechanisms (blink, timewalk, dagger) are going to allow them to get away from most gank attempts. Understanding this, a gank would ideally involve a third hero circling into the lane to stun/disable first (unexpected) and then following up with the 2nd and 3rd from the heroes already in the lane stacking the disable duration for a kill and to prevent the blink out.

Tri-Laning (3 heroes in your lane against Anti-Mage from the start) - is a decent option as well, all 3 heroes need to have a stun, disable, or a decent slow skill for this to be appropriate. You're looking to press hard on the enemy heroes and push through their towers when they linger back. The general idea being heavy pressure and pushing down towers early and getting in the way of the usual progression a carry needs to farm/level.

Heroes

Lion - an excellent ganking hero early game (even before level 6), leveling hex and stun, coupling Lion with any other hero (or two heroes!) is an easy combination for early harassment. Lion should also be looking to incorporate a blink dagger into his build sooner than later, to quickly get the drop for ganks (few people stand around as Lion charges at you) and also in order to keep up with Anti-Mage when possible.

Blink in, hex, wait, wait, stun, wait wait. This is a long time to be disabled. You can blink to chase as well, but the key here is to get in his face before he realizes stuns are incoming so ideally you are blinking in and disabling before he can blink out.

(Rhasta could potentially be an option, though less recommended as his second disable, Shackles, requires him to be in the line of fire and also immobilized for the length of the disable, making it less reliable to get the full duration of shackle time)

Spectre - if you're looking for a late game carry that will stand up to Anti-Mage when a game has gone 50-60+ minutes. The problem here is that you are relying on a number of other factors: [1] that you can out-farm anti-mage, [2] the enemy team allows you to get to late game without too much gank interference, [3] the enemy team does not push through to your barracks before your core items are farmed.

Outworld Devourer is the Anti-Anti-Mage. Anti-Mage. In practice AM's skills have excellent synergy with each other. Draining mana until he can ulti you. But OD has essence aura. AM's ulti is basically useless now. Also his resistance to magic is very misleading towards OD. Although it's technically magic based it's pure damage meaning it goes through all resistances. Only magic immunity can shut OD down, but building a bkb isn't something an AM wants to do. And as for his blinking bullshit do not worry as Scythe of Vise is a very very common item on OD. And if evasion from a butterfly is keeping you down a hex plus the pure damage will surely bring him low. Then after that just imprison him and finish him off with Sanitys Eclipse.

Items

Scythe of Vyse - nothing too surprising on this recommendation, hopefully. Later in the game you're looking for a sheep-stick on one or two heroes to help with a few more Hex casts as Anti-Mage becomes more difficult to kill outright.

Other items - most other items are going to focus on dealing damage, because while mass disables are good, you still need your carry/semi-carries to start taking down his HP (not necessarily with nukes).

It is not usual for 2 to go mid also because the mid position offers you the possibility to go gang on the up-lane or down-lane very fast. It is not as effective when there are 2 mid players `cause they get half XP each and they reach hard lvl 6 to go on effective gangs.