Learn English – “The crying baby gets the milk”

Etymonline says

For Pete's sake is attested from 1924, probably a euphemistic use of the disciple's name in place of Christ;

There is a kind of a pop evo-psych explanation lurking here. Lovers are called "baby," because they tend to evoke the same kind of feeling one experiences with a baby. Babies are "designed" to do this; they need to excite the same passions as lovers to be taken care of, especially human babies, whose species are huge outliers in the amount of parental (and paternal) investment they require among the mammals.

If one wants to be cold and mechanistic, one might say that babies and lovers exploit the same neurological circuits; the wires are quite literally crossed in the brain. Because evolution most often proceeds by co-option of existing behavioral repertoires, this make sense; natural selection can only search within a small neighborhood of a population's current fitness, i.e. a small neighborhood of its members' current behaviors, within the fitness landscape. When we underwent the transition from our ape-like ancestors, who probably lived multimale-multifemale social groups like their descendants the chimpanzees, and whose sexual competition was similarly intense, to our derived strategy of long-term pair bonding and paternal investment, it is a strong hypothesis that babies must have particularly exploited the emotional circuits that deal with pair-coupling and sexual attractiveness. Here is an in-depth treatment of this hypothesis by a certain David Brin.

Thus, as both babies and lovers evoke similar feelings, it makes sense that the words associated with these feelings would be in some sense interchangeable with one another, though I wouldn't extrapolate that understanding too far.

I'd also venture that "baby" as a pet name is directed much more by men towards women than vice versa, though I have no direct evidence on this point as yet. This is because neotenous features, such as large eyes, fine hair, and high voices, are important in determining women's attractiveness, as their age really matters with respect to their expected lifetime fertility, the ultimate metric that natural selection evaluates. Notice that babies share many of these same attributes, which are lost in puberty in men through the action of the "master" hormone testosterone. Men put much more of a premium on neoteny and physical attractiveness in general, so you would expect a difference in men calling their female lovers "baby" as compared to vice versa across cultures.