I'm no expert at Java, so this isn't a decisive answer, but I figure it can't hurt to post the code here anyway for others to decipher. From SpawnerAnimals.java:
(Comments mine)
private static boolean canCreatureTypeSpawnAtLocation(EnumCreatureType enumcreaturetype, World world, int i, int j, int k)
{
if(enumcreaturetype.getCreatureMaterial() == Material.water)
// If it's a water creature...
{
return world.getBlockMaterial(i, j, k).getIsLiquid() && !world.isBlockNormalCube(i, j + 1, k);
// Make sure it spawns in water
} else
{
return world.isBlockNormalCube(i, j - 1, k) && !world.isBlockNormalCube(i, j, k) && !world.getBlockMaterial(i, j, k).getIsLiquid() && !world.isBlockNormalCube(i, j + 1, k);
// Makes sure the block it's spawning on is opaque, that it's not water, and that there are at least 2 blocks of air overhead.
}
}
It does not appear that it checks whether the mob is spawning underground or not.
Anecdotally, wandering around for a while with an Xray texture pack produced only surface mobs, so I'm not sure what other guards might be put in place to prevent underground spawns.
Best Answer
Passive mob spawning has changed in version 1.8. Animals now spawn when a new chunk is generated and they don't despawn. However, there seem to be some disagreement as to what happens if you kill all the animals around you. Some say that they don't respawn while others claim that they do but do so very slowly.
The best thread I've seen about this is this one posted mere hours ago. It seems to confirm what you see, that animals don't respawn at all. UPDATE: The same thread now seems to confirm that they do respawn, just very slowly.
Also, I think everyone agree that animals don't spawn in ocean biomes at all. I have played on a couple of islands far out to sea myself and have never seen animals there.
In version 1.9 breeding of animals will be introduced. Hopefully that will clear things up.