As @Unionhawk said, your mobs are almost certainly de-spawning.
This means they vanish from the world forever, like footprints in the sand.
But as time goes by new mobs spawn, such is the circle of life.
In order to have a proper 'zoo' of mobs, you need to stop them de-spawning. In the case of Zombies, (as @MrLemon pointed out) simply throwing an item at them and letting them pick it up is enough to keep them spawned. However this only works for Zombies.
Thankfully, as of 1.6.1, there is an item designed to both name mobs and keep them spawned for good - the name tag.
Name tags can be found naturally within chests that spawn in dungeons or (as of 1.7.2) by fishing.
If you're in creative you can pull one straight from the creative inventory, and if you have access to console commands (i.e. you allowed them in map options when creating the map) you can use this command:
/give <player> 421 <amount>
Where <player> is replaced with your minecraft name and <amount> is replaced by the number of items you want. For example I would type:
/give Pharap 421 1
which would give me a single name tag.
Alternatively (as pointed out by @SaintWacko) you can avoid specifying your name by typing this instead:
/give @p 421 1
Also note that an enderman is the only mob naturally capable of teleporting, thus even after naming one it is still capable of escaping. Also note that name tags do not stop mobs from dying, so try not to suffocate them or set any on fire.
If you have access to commands, you can use
/effect [username or @a (for all players)] 23 99999 10
Effect number 23 is the effect for saturation. It will replenish your hunger bar. With the power of it being set to 10, it will regenerate half of the whole hunger bar per tick (you can raise it to 20 for the entire bar at a time).
To refresh it, you just need to type in the command again. If you use Command Blocks, you could easily set up a button to do it (if you have access to commands).
Best Answer
Fencing works very well in this case, as mobs cannot jump over fences, with the exceptions of the Spider.
Remember, you want a lot of lighting around your house. Hostile mobs only spawn in dark environments, so you want plenty of lighting in and out of your house to avoid the chance of a mob spawning or getting close to you.
You can also try building lava traps, like a moat, so mobs that try to get too close end up getting burned as a punishment.
In general, it's all about lighting and keeping the area around your house open and free of darkness.