The composition of your team is the first line of defense against Anti-Mage. Having a good core of gank-tasked heroes (3+) and ideally, no more than 1 hard carry. With Anti-Mage, like many of the late-game carries (Spectre, Void, Antimage), your game should not fixate on countering him late game, but instead ask what counters Anti-Mage?
Wards, Ganks, and Pressure
You need to shut down these hard late-game carries early. Even allowing the completion of a single core item uninterrupted is a bad sign. Where you listed 3 high tiered items, the momentum for Anti-Mage was likely far out of your control to rein in late game.
So you're looking to gank him frequently. Note that if the enemy team has more than one late-game carry this becomes more difficult for both teams, where your team will need to keep both late-game heroes in check and from the enemy team's side they need to [1] not die, and [2] wait you out. So logically, you need to do the opposite, gank them far more often than you would consider necessary and push down towers.
While "kill them and push towers" is a bit too simple-minded of an approach, the crux of the issue is defined.
The problem is that in execution, as is commonly the case especially on randomly assembled teams, not many players risk leaving their lane, leaving behind their potential farm-able gold, and losing experience roaming from one lane to another.
But what is important here is to reframe the issue, again, and that is a successful gank does not always need to result in a kill. Pushing Anti-Mage out of the lane to hug the tower, deny his gold/experience gain, return to base to heal, or even just added pressure that you guys are coming for him stacks up for an effect in your favor. Understanding that 'yes he has blink' and 'no you probably won't kill him' you are primarily focused on disrupting his acquisition of gold and experience.
Laning against Anti-Mage should involve 2 heroes that can stun or disable, understand when picking a lineup against Void, Anti-Mage, or even Spectre, their escape mechanisms (blink, timewalk, dagger) are going to allow them to get away from most gank attempts. Understanding this, a gank would ideally involve a third hero circling into the lane to stun/disable first (unexpected) and then following up with the 2nd and 3rd from the heroes already in the lane stacking the disable duration for a kill and to prevent the blink out.
Tri-Laning (3 heroes in your lane against Anti-Mage from the start) - is a decent option as well, all 3 heroes need to have a stun, disable, or a decent slow skill for this to be appropriate. You're looking to press hard on the enemy heroes and push through their towers when they linger back. The general idea being heavy pressure and pushing down towers early and getting in the way of the usual progression a carry needs to farm/level.
Heroes
Lion - an excellent ganking hero early game (even before level 6), leveling hex and stun, coupling Lion with any other hero (or two heroes!) is an easy combination for early harassment. Lion should also be looking to incorporate a blink dagger into his build sooner than later, to quickly get the drop for ganks (few people stand around as Lion charges at you) and also in order to keep up with Anti-Mage when possible.
Blink in, hex, wait, wait, stun, wait wait. This is a long time to be disabled. You can blink to chase as well, but the key here is to get in his face before he realizes stuns are incoming so ideally you are blinking in and disabling before he can blink out.
(Rhasta could potentially be an option, though less recommended as his second disable, Shackles, requires him to be in the line of fire and also immobilized for the length of the disable, making it less reliable to get the full duration of shackle time)
Spectre - if you're looking for a late game carry that will stand up to Anti-Mage when a game has gone 50-60+ minutes. The problem here is that you are relying on a number of other factors: [1] that you can out-farm anti-mage, [2] the enemy team allows you to get to late game without too much gank interference, [3] the enemy team does not push through to your barracks before your core items are farmed.
Outworld Devourer is the Anti-Anti-Mage. Anti-Mage. In practice AM's skills have excellent synergy with each other. Draining mana until he can ulti you. But OD has essence aura. AM's ulti is basically useless now. Also his resistance to magic is very misleading towards OD. Although it's technically magic based it's pure damage meaning it goes through all resistances. Only magic immunity can shut OD down, but building a bkb isn't something an AM wants to do. And as for his blinking bullshit do not worry as Scythe of Vise is a very very common item on OD. And if evasion from a butterfly is keeping you down a hex plus the pure damage will surely bring him low. Then after that just imprison him and finish him off with Sanitys Eclipse.
Items
Scythe of Vyse - nothing too surprising on this recommendation, hopefully. Later in the game you're looking for a sheep-stick on one or two heroes to help with a few more Hex casts as Anti-Mage becomes more difficult to kill outright.
Other items - most other items are going to focus on dealing damage, because while mass disables are good, you still need your carry/semi-carries to start taking down his HP (not necessarily with nukes).
Personally I found that a great way to improve at the game is to constantly learn new heroes. This gives you a great insight into how different heroes play at different stages of the game, and with different items, along with how they lane in different situations, and who they partner up well with; as well as keeping the game fresh and interesting (variety is the spice of life!)
For example - I kept getting beaten by Axe, so played a few games with him, did ok, and then watched the 'Purge coaches' video where he helped someone on Axe, which improved a) how I play Axe and b) how I play against Axe.
And this is something I now regularly do: get beaten by a hero/watch them played in a pro game, so I play one or more bot games depending on complexity of said hero (i.e. lion/lich - one game. Meepo/brewmaster - 3 or 4 games), watch a Purge video, and then dive into a real game. If I like the hero, I keep playing them, but if I don't, at the very least I've understood that hero's mechanics and playstyle, and over time, this will help you build a strong knowledge of the heroes, their builds, and item use.
If interested, my dotabuff ID is 98862191 "Cjeesebar" - you'll see I'm learning PA just now :) having just finished learning Bristle, LC before him, and WK before him; bit of Brew mixed in there too. Happy to take any questions on this approach.
Best Answer
First off, dota does not balance games so your probability to win is defined at 50%, this is a common misconception. Rather, your probability to win is 50% based on your - and your teammates / opponents - past performance under several assumptions, most of which won't be true in a real-world scenario. E.g. If you decide to pick nature's prophet, farm up a rapier, and teleport into the opponents' fountain it's likely to be much lower (and earn you somewhere between 3 and 10 reports). The system won't account for this. Thus the 50% general rule is more of a natural consequence of the rating system, and based around the assumption none of the players in the game is over- or underrated. Dota balances games with the following goals in mind:
The matchmaker will try to achieve these goals as best it can, with goals towards the top of the list being the more important ones. For example very high or very low MMR will increase your queue time as the system will take more time to match you with suitable opponents and teammates.
Yes, it is possible to alter this score significantly. The very simplest way to imagine this is if you were to swap your account with someone who is 2,000 points below you. At this much lower rating, you're likely to win 90% of your games or more, which will steadly increase your rating.
For most players, getting better relative to the general populace will gain you some rating points, as well as the converse.
Unless you pick up and retain a lot of bad habits from this experience your rating should fairly quickly get back to its original level.
MMR is roughly based on the Elo rating system.
Recently, someone did a Survey of players' MMRs. Currently, the distribution has somewhat more of a long tail to the right. Also, this survey has an average that is quite a bit higher than the real mean for dota players (which is around 2,250) because it has a natural bias; higher skilled players are more likely to register their MMR with external parties.
The real curve will look quite similar, but with a centre around 2,250 and similar standard deviation.
The explanation for the 'long tail' to the right is due to dota being a team game and there being a 5-point minimum gain after a game. Essentially, to explain the former, what will happen is that the game will match players that are at say 8,000 MMR with ~4,000 rated teammates against ~4,800 rated opponents. However, in general the positive effect of having one (or two) really good players outweighs having worse others (it's the nature of the game). These two effects create a long tail among the ~1,000 best players while one really would not expect anyone to exceed about 5,500.
The five-point minimum coupled with no 'limits' as to the maximum diffrence in skill between the two teams means that extremely unbalanced games give the team that was expected to win too many points if they do, according to the classical algorithms.
Even so, there comes a limit where a game is literally unwinnable / uncarryable, even for someone like for example w33. Whatever MMR delta between the other players that is is where the maximum lies. (My guess is around 2 standard deviations, times 4 (4 others), or 4,800 points above the regular maximum, so somewhere between 10,000 and 11,000).
For additional information, you can visit the official wiki articles on Matchmaking and Matchmaking Rating.