Minecraft Java Edition – How to Breed a Polar Bear

Feed both animals their specific food at same time, while they are beside each other. That will make them have hearts above their heads and breed with each other.

• Sheep: wheat
• Cows: wheat
• Pigs: carrots or potatoes
• Chickens: seeds, any type (wheat, pumpkin, melon, or even Nether wart)
• Horses: Golden Apple or Golden Carrot
• Tame wolves: bones (to tame). Rotten flesh and any form of meat, cooked or raw
• Tame cats/ocelot: raw fish (sneak, holding fish so it walks up to you, then tame)
• Turtles: seagrass (can be obtained with shears)
• Dolphins: raw cod
• Polar bears: fish (any kind)
• Pandas: bamboo
• Parrots: seeds

This applies to Minecraft versions 1.8.0 - 1.13.x

The 1.14 Village and Pillage update changed breeding mechanics again

That your Villagers are entering into "mating mode" (i.e. they occasionally have hearts floating out from their bodies) is a good sign. It implies that the Villager's surroundings are suitable for breeding to occur.

However, with the 1.8 release Villager breeding has been tweaked. Villagers now also have to be "willing" to breed, being in "mating mode" isn't enough anymore.

From the Minecraft Wiki:

Willingness

Additionally, villagers must be "willing" in order to breed. After mating, they will no longer be willing.

Villagers can become willing by the player trading with them. Willingness is granted the first time a new offer is traded, or at a one-in-five chance on subsequent trades. This will not cause them to immediately seek out a mate, however.

Villagers can also become willing by having either 3 bread, 12 carrots or 12 potatoes in one stack in their inventory. Any villager with an excess of food (usually farmers) will throw food at other villagers, allowing them to pick it up and obtain enough food to become willing. You can also throw bread, carrots, or potatoes at the villagers yourself to encourage breeding.

That would suggest that it's this new "willingness" mechanic that's actually holding your village back. As the article suggests, you should be able to make the Villagers more "willing" by throwing enough food at them. A caveat though: don't expect this to be quick when your initial population is small!