Enforce the Handle Animal Rules
Getting those dogs to do what they want requires a Handle Animal check (DC 10 and a move action if it's a trick). Each PC can only do that to one animal at a time. Otherwise, the dogs will just generally do whatever you as the DM want them to. That might mean they all swarm something, or it might mean they find the target frightening and run away from it.
Remember, animals will only normally attack other animals, humanoids, monstrous humanoids, and giants. The animal has to be specially trained and use two tricks to have an attack command against anything else.
Kill The Dogs
Unless these are advanced animals with bonus HD, Dogs have 6HP. At 5th level, one Fireball will wipe them all out (the Riding Dogs you gave the party have 13 HP, so their survival chances are better but it should still kill some of them and make the Handle Animal checks harder on the others due to being wounded), and since Dogs are pack animals and one PC can't direct five dogs to attack five different things in one turn, they're likely to not be that far apart.
A Barbarian with Cleave could make short work of them as well. I'm not saying to go far out of your way to kill them, but they're low HP targets in combat, and they make a good target to even the numbers in a fight for certain types of enemies. If none of them are dying in combat at this level, it would be pretty strange.
You could pretty easily make an encounter that would eliminate the dogs in a turn or two, at which point the party will have to decide if it's really worth spending money to buy new ones.
Alternately, you could kill one or two of them in an encounter (which should happen normally in EL 5+ encounters), and see if the party decides that having dogs that aren't up to higher end combat around is a good idea. Any animal lover characters wouldn't want to put animals in harms way that aren't up to it, and the more frugal characters wouldn't want to have something die when they could sell it for gold and get more value out of it.
Initiative Shortcut
When this happens in my game, I roll one initiative for all of a given type (all the Dogs in this case), and have them all move at the same time. That cuts down the annoyance a little bit.
I also don't put much thought into what they're doing. Dogs are animals, not tacticians. They will either do exactly what they're ordered to do, or they'll attack, or they'll not attack. They're not going to get fancy figuring out how to reposition themselves for optimum protection of the PCs. (The PCs would have to use Handle Animal commands to do that.)
As they only have one attack, the rolling should be pretty limited and so you can keep the pace up by acting quickly without thinking about what they're doing.
Party Logistics, As You Mentioned
Is the party carrying enough food for five dogs? Are you enforcing that? Caring for five dogs in real life is actually quite a lot of work, make them deal with that.
Up to the DM
There are no official general guidelines, as far as I know, for that mechanic. The only non-magical thing anyone can attempt is the skill Animal Handling (Wisdom).
As any other skill, it's treated mostly as a binary result - either you fail that skill, or you pass that skill check. Note that, by the PHB, Animal Handling can only be used to calm down domesticated animals. Allowing anyone befriend a wild, hostile animal would make Animal Friendship a lot worse (and it's already not amazing).
One example of using that for calming down seemingly wild animals is given in the Lost Mine of Phandelver adventure from the starter set (spoiler ahead)
In the first dungeon described in the book, there is a kennel with 3 wolves. Anyone can try to calm them down with a DC 15 Animal Handling check. If you give them food, the DC is reduced to 10 instead.
Again, it depends - somehow - on the animal itself. A hungry wolf might be calmed down by giving it food, while a bear trying to protect its little baby-bear might be calmed down by simply walking away and showing, somehow, you are not there to harm their child.
Similar to my answer in your other, similar question, it depends on the NPC's (and here I'm saying NPC as any character, including animals) motivations and personalities for the encounter.
Best Answer
There are no general official rules for taming wild animals (related)
Slight aside:
It should be noted that the Animal Handling skill gives a few examples of its usage, such as calming a domesticated animal, keeping a mount from getting spooked, intuiting an animal's intentions and so on. Emphasis mine.
These are only examples, but the skill clearly has limits. It seems reasonable that a DM may allow it to be used to temporarily calm and ward off a wild animal but "taming" a wild animal to the point it can be commanded in combat is far beyond this skill's parameters.
And, despite this being a fantasy game, using an ordinary skill to tame a wild animal isn't particularly feasible either in any realistic sense (it typically takes generations of breeding to get from a wild animal to a tame one, let alone one that will fight for you).
But mainly from a balance point of view, this would (a) be particularly overpowered if it effectively gives a character additional attacks over and above their normal and (b) steps on the toes of classes designed around this feature (such as a Beast Master ranger you noted and a druid's Summon Animal spells).