When it comes to pairing specific ingredients, the general thinking seems to be that ingredients that share "flavour notes" in significant quantity will go well together.
In practice it goes much deeper than that, because in addition to taste, foods have very specific and recognizable aromas which affect the perception of flavour; wine is mostly sweet and sour, but you wouldn't substitute ordinary grape juice for it; the aroma of the alcohol is unmistakable, and even between different wines you will have varying levels of "fruitiness" or "woodiness".
But let's start with tastes:
Chocolate and coffee go well together because they share characteristic bitter notes. The roasting of the beans also makes a difference; the pairing works best when the roasting is enough to thermally degrade the bitterness without losing too much sweetness due to caramelization. This emphasizes the sweetness and de-emphasizes the bitterness and acidity of the coffee, making it closer to the flavour profile of chocolate.
A traditional stock combines a mirepoix (onions, carrots, celery) with meat (well, bones), both very high in umami. The end product may have varying notes of bitterness or saltiness from the vegetables but is basically one huge kick of umami. A lot of people also enjoy peanut butter and bacon together, which combines the significant umami and saltiness in both foods.
Beets and sour cream are a traditional (or so I'm told) Polish dish, both obviously sharing strong sour notes. Often chives are added, which are mildly pungent in nature and pungency tends to go with sour or bitter (since it is created by sulfur which is perceived as sour, bitter, and also "metallic").
This tends to work reasonably well for basic pairings, but another very strong element of this is balancing all of the different flavours in a dish.
Peanut butter and jam don't seem to have much in common, flavour-wise. But when you combine them together in a sandwich you have the high umami and saltiness of the peanut butter, the sweet and slightly sour notes of the jam, and the hints of bitterness and/or sourness in the bread. Together you have strong notes of all five tastes and the result is appealing to most people.
Five Spice is usually a mixture of cinnamon (sweet), star anise (mildly sweet and bitter), fennel (mildly sweet and fairly savoury), ground cloves (pungent and somewhat bitter), and Szechuan peppercorns (salty and pungent with a sour aftertaste). The quantities are generally adjusted to balance out the tastes; for example, fennel is the only really savoury ingredient so Five Spice tends to have more of that than other ingredients. Anyway, this is used all over Chinese cuisine as a sort of "wonder spice".
A really good roast chicken starts off high in umami and is then brined (salty), buttered (sweet), and roasted which involves the Maillard reaction (bitter). This is missing sour, which is why a lot of cooks will throw a few lemon slices in there.
But truthfully, even though this all sounds plausible, it's really so much more complicated than that. Even though there's some science at work here, the five tastes are a little like the four elements or humours; it's an almost archaic way of thinking about food chemistry, because our mouths and noses can detect so much more than that.
If you want to see just how little we actually really know about flavour today, check out They Go Really Well Together on Khymos, linked to from one of the related questions. There you'll see totally nonsensical combinations like chocolate and garlic or mint and mustard that actually - with the right preparation of course - produce pretty pleasant flavours/aromas.
The truth is people are still gathering data on this and we don't really know, exactly, what makes some flavours work together. At least we're starting to approach this scientifically and actually experiment and document this stuff, but there's not enough data to form a coherent theory yet. Check the answers to my question about flavour pairings and the other question on pairing resources if you want detailed data on which flavour pairings can "work", although you might not get much of an explanation for why.
Per AsianResearch.org's article The Ancient Theory Behind Chinese Food:
Generally
speaking, foods that have a higher water content are considered cool,
or yin, in nature. These are often foods that are boiled or steamed.
Foods that have a higher energy content, particularly from fat, are
considered warm, or yang, in nature. These are often foods that are
fried or roasted.
[...]
From [Chinese traditional element theory] comes the idea of the five flavours −
bitter, sweet, spicy, salty and sour.
These flavours are subdivided into yin and yang. Sweet and spicy foods
are considered to be yin, while bitter, sour and salty foods are
considered to be yang. The five elements also correspond with the
colours red, yellow, white, blue and green, which are all considered
when choosing ingredients.
Best Answer
Developing upon a previous idea from @mroll, eggs + fat (oil or butter) has some interesting science to explore.
By whisking the egg white, you incorporate air into the liquid phase and form a foam. The science behind is that of protein denaturation. More here. A lot of recipes rely on the stability of this foam to create a 'light' texture. The simplest examples is a soufflé. You carefully mix the yolk back without breaking the foam and place the mix in a ramekin. In the oven, the air will heat up and tend to scape upwards (hot air go up). Meanwhile, the denaturated proteins will coagulate due to the heat and get stiff. The interplay between the air pushing the foam up and the proteins coagulating is the reason why the temperature in the oven (not too hot, not too col) is crucial in preparing soufflé.
The proteins in the yolk and white coagulate at different temperatures. If you simply mix all together and put in a very hot fire, you simply coagulate the whole ensemble of proteins and get a dry scrambled egg / omelette. However, you can play with this difference of temperature to create new interesting textures. Get a bowl with your yolk+white mix and put on the top of a pan with boiling water (heat bath or bain marie). Whisk the mixture continuously, so that you get a better control of temperature changes. You will notice that at some point the texture will start to thicken, while still liquid. At this point remove from the heat: you got a cream! If you salt it and add some fat, you get a hollandaise-like sauce. If you add sugar, you got a custard.
Of course the above is only for the sake of science. To get flavourful dishes should add other ingredients such as salt to the mayonnaise, cheese to your soufflé, etc.