Anti-Mage is a strong carry and Shadow Shaman a strong disabler/pusher. I assume the two of them are laning together, but if not the same will still somewhat apply. Trying to outcarry Anti-Mage usually isn't a very good prospect. If you're really determined to try to do that, Riki and Slardar are your best options.
Otherwise, you need to have someone strong to harass Anti-Mage when he's at his weakest and slow down his farm. If he's allowed to farm easily for 10-15 minutes he's incredibly harder to deal with in teamfights. You can't expect to kill him, however: blink is too good of an ability. Instead, you want someone who's not going to be threatened by him much in lane.
Shadow Shaman is not a very good babysitter because he needs early levels. His autoattack range, damage, and armor are all low, and so he is easily outclassed by an enemy harasser. Once Shadow Shaman is more subdued, you can turn to keeping the Anti-Mage from getting easy farm.
I recommend Lich or Venomancer for this purpose, but you'll need a lane partner with a stun as well to prevent Shadow Shaman from just gripping you. Anyone with a stun will do, but I'm not sure if the bots will be smart enough to use their stun as an interrupt. (At this point I should mention that you guys should be laning against each other if you're going 2v2 with 3 bots per team, otherwise you're just seeing who's better at taking advantage of dumb bots.) Other heroes like Shadow Demon, Obsidian Destroyer, and Bane can be useful counters to Anti-Mage at various stages of the game because of their spells, while heroes like Leshrac and Death Prophet can push towers and make it more difficult for Anti-Mage to farm.
Countering Rhasta (Shadow Shaman) is a lot more simple. You just need to be coordinated to quickly stop his Shackle, and don't be out of position enough to be Hexed. If you get ward-trapped, either use Phase Boots to walk out or kill one of the edge (not corner) wards and then walk free.
Ghost Scepter and Force Staff deserve special notice as great item pickups against both of these heroes. Shadow Shaman's wards won't be a threat and Anti-Mage can't stay on you (but be wary of his allowing his ultimate to do extra damage with Ghost Scepter). Force Staff also helps save teammates (Force Staff them away from the Anti-Mage, forcing him to commit more heavily for a kill attempt).
tl;dr: Lane Lich and a stunner against the AM. Get pushing power elsewhere. Deny him farm, use Frost Armor/Force Staff/Ghost Scepter in midgame to prevent him from killing people quickly with melee attacks. Push heavily during this time to earn a gold lead and hopefully take a set of Barracks quickly. You don't want to go lategame against an Anti-Mage without a similarly powerful carry like Riki/Slardar/Faceless Void.
Your example lineup is viable and can win depending on which tatic is going to be played and player skill of course. However your lineup is forced to go an all in tactic since it will not outcarry the second lineup.
If they can manage to push one lane of rax around the 20 min mark they have a solid chance of getting another lane and win the game around the midgame. Since they have a solid global ganking strat combined with push it will be hard for the enemies to get a solid early to midgame.
But if they play with mapawareness and defensive they will not get picked of too much and stop the first rax push since they could farm their midgame items on their carries.
So in conclusion: Your lineups are not a good example of extreme cases of lineups and therefore can both win.
There are lineups that are unlikely to win. Those are present in AtoD tourneys (DotA reversed) where people pick the lineup for the enemies (making it extremely bad).
For example an 5 hero lineup with no early game presence like spectre, riki , etc.
Those will most likely not win vs a rounded or a strong early game team.
And to answer your final question: There is no lineup for a guaranteed win if we are talking competitive CM mode. If you are talking about AP there are good synergy lineups but nothing that would not be counterable. Dark Seer + Naga Siren + any more aoe heroes was once considered extremely hard to beat and navi did it with a fast bkb on enigma and an naturally magic immune hero like juggernaut.
Final conclusion:
Picking lineups in CM mode is taking away a lot of the game already. Forcing enemies into tactics that they expect to beat is an extreme advantage.
Best Answer
The reason bots only select a limited set of heroes is because the AI hasn't been scripted for every hero in the game yet. Assuming you pick before them, the bots will then try to counterpick any heroes you or your allies have.
You can also 'suggest' heroes which will cause allied bots to choose those heroes, if AI exists for them.
As far as I can tell, the only AI scripted heroes currently are:
Valve periodically adds new bots in their weekly updates, but as of late they have been focusing primarily on AI enhancements instead of adding more heroes.