[RPG] Does making an animal more intelligent automatically make them a magical beast

You're trying to use a D&D precedent in Pathfinder, but the Diplomacy rules were specifically changed!

The line "You cannot use Diplomacy against a creature that does not understand you or has an Intelligence of 3 or less" was explicitly added to the skill description in Pathfinder; it was not present in prior incarnations of the rules. Given this, it hardly matters whether it worked in 3.5 or not.

what is this witch ability good for if it doesn't allow for using diplomacy?

Um, talking to animals, same as the spell it emulates? You can still get information from them. I believe you could also cast [language dependent] spells on them, if you want a more specific game benefit.

Augmented [original type] as a subtype has the following description:

A creature receives this subtype when something (usually a template) changes its original type. Some creatures (those with an inherited template) are born with this subtype; others acquire it when they take on an acquired template. The augmented subtype is always paired with the creature's original type.

It doesn't say anything about the creature continuing to be or count as a creature of its original type, and primarily seems to function as a way to remember what this creature was before it changed types. The augmented subtype is typically applied when some part of a creature's statistics remain based on its old type: for example, a vampire does not recalculate HD, hp, or BAB when becoming undead; additionally, the vampire's create spawn ability only works on creatures of its original type. A ghoul, however, despite being a former humanoid turned undead by an undead creature's special ability, exactly like a vampire, does not gain the augmented subtype. A ghoul's statistics and traits, though, are not based on the original creature's; not even the creature's race matters except perhaps to help determine appearance. A ghoul's stats are set; there's no "ghoul template" to apply.

The rules for animal companions feel the need to call out companions as "remaining creatures of the animal type for the purposes of determining which spells can effect them", although animals whose Int score increases to 3 normally become magical beasts (augmented animal). This implies that allowing the creature to change to a magical beast (augmented animal) would disqualify it from having spells like Animal Growth cast on it. Indeed, the Awaken spell on the Pathfinder SRD specifically mentions that creatures targeted by it can no longer serve as animal companions. Awaken causes them to become magical beasts (augmented animal), while an animal companion must be an animal.

After looking at which creatures do and don't gain the augmented subtype, I'm reasonably certain the correct use of the subtype is: Apply the subtype if the creature's statistics are partially based on what the creature was before becoming its new type, or if it has abilities that rely on recalling its original creature type. I'm sure that the original type no longer counts as its current type; a creature can only have one type. Part of the benefit of class archetypes that change your type is that you become immune to the many humanoid-only control spells, such as Dominate Person or Charm Person, and part of the drawback is that you can no longer be affected by humanoid-only buffs, such as Enlarge Person. Otherwise the type change would be near-meaningless, and interpreting the rules in situations where one of your types is specifically vulnerable to an effect and the other type is not would be ridiculous.