Minecraft – What shapes can Nether Portals be

This is how it works:

Every time you enter a portal, the game looks for an exit portal inside +/-128 blocks square (y is irrelevant). If you enter a portal at x=100, y=60, z=200 in the overworld, this corresponds to x=12, y=60, z=25 in the Nether. The game scans a square from (x=-116, z=103) to (x=140, z=153), for all y values from 0 to 128. The closest portal in that space is where you appear. If there is no portal in that area, one will be created in a suitable place. Since there is a chance the x=12, y=60, z=25 will be obstructed, the game will search for an open space in that +/-128 blocks square. If such space is found nearby, all will be good, the new portal will lead back to the same one in the overworld.

Note that the game will only scan within the map height. That means that if you place a portal above the nether ceiling, it won't be found, and the game will create a new one for you.

However, sometimes the game will put the Nether portal far from the starting point, if it can't find other suitable place. Let's say it puts the Nether portal at x=80, y=60, z=110 (this is still in the +/-128 bounding box). When you enter that portal the game will search the corresponding space in the overworld: starting from x=640, z=880, it will search the +/-128 blocks - from x=512, z=752 to x=768, z=1008. As you may notice, the original overworld portal is well outside this box. So the game will create a new portal in the overworld. This is what is happening in your world.

To fix that, write down the coordinates of the portal in the overworld (use F3 to get them) and divide them by 8. Enter the Nether, go to the calculated coordinates and create a portal from the Nether. You don't have to be exact, as long as you are within 16 blocks from the calculated coordinates (you need to match only x and z, y is irrelevant). Then the game should find the original overworld portal. Or: Move the overworld portal using the above logic.

In described situation, there is no way to make the two portals lead to each other without moving either of them. I'd move the nether portal and if the new position is outside of the fortress, I'd build a short safe walkway to the fortress.

Here is a crude drawing of the process:

According to the Minecraft Tumblr, this has been fixed for 1.4.4. If you haven't updated or can't due to the server your on, or are still having problems there may still be some solutions.

One thing you can try that I've seen having at least limited success, is dig a pit around the portal in the nether, such that it would be impossible for the animal to jump back in. It looks like you may have tried this already, but I don't think it was deep enough (at least 2 blocks down from the edge of the portal).

If this is on a server, another option to at least try (I have no evidence that it'll work) is to have another player waiting on the other side with wheat (or another appropriate luring agent) so that the animal is lured away from the portal as soon as you teleport them.