Baking powder, especially if too great a quantity is used, adds an unpleasant flavor to a baked good. Even in an appropriate quantity it can be noticeable and it certainly doesn't do anything to enhance the flavor. Many baked goods traditionally don't use a chemical leavener at all, but instead rely on technique. Creaming butter and sugar together or whipping egg whites was historically used to make cakes which rose solely based on the bubble network that was created.
Yeast, on the other hand, creates a delightful flavor that you associate with your favorite crusty loaf of bread. Yeast can be used not only in making bread but also in some excellent cakes (St. Louis Gooey Butter Cake, for example, although many "knock off" recipes cheat here and miss out on the true goodness). Yeast also provides a significantly different texture during the rising due to the intentional creation of a gluten network (usually something you absolutely don't want in a quick bread or quick cake) — you don't get a crumb with big, airy holes from baking powder or baking soda with an acid.
Also, in my experience baking powder lasts six months in the pantry and yeast lasts at least six months in the fridge. The shelf life is not so different.
If yeast scares you, you may want to check out some of the proponents of the no-knead bread technique.
For a lot more information on this subject, there is a recent publication that covers all sorts of leavening agents.
The main problem with adding the baking powder last would be getting it evenly incorporated throughout the dough or batter.
In the traditional methods where it is in the dry ingredients, it can be sifted or whisked evenly throughout the dry mixture which facilitates having it evenly distributed in the final batter.
If you tried adding the powder to a batter that was prepared except for it, even sprinkling it across the surface, it would be more difficult to evenly distribute throughout the entire batter. Also, you would then have to mix or beat the batter again to distribute the baking powder, which would develop additional gluten and lead to a tougher end product.
Remember, baking powder requires liquid in order to react (in dry form, the acid and sodium bicarbonate don't interact with each other). When you think about it, in the major baking methods where baking powder is used, the baking powder is activated as late as possible right before baking—the wet and dry parts of the batter don't meet up until the last step:
Muffin method. The wet and dry ingredients are mixed separately, and can be held. They are combined, mixed gently, then portioned into molds or pans and baked immediately.
Creaming method, common for cakes. Typically, the ingredients will be divided into the three groups: an emulsification of creamed butter and sugar, eggs, and perhaps vanilla; a liquid ingredient or mixture (such as sour cream or milk), and the dry flour mixture. Each of these stages again can be set up in advance. They are combined (usually alternating the liquid and dry in two or three stages to maintain the butter/liquid emulsion and to minimize lumping) as the last step in developing the batter.
So even if you need to try to fit baking in with other preparations, you can stage your prep, up to the final combination at your leisure.
If I want muffins for a brunch party (or cornbread on Thanksgiving, when my oven is scheduled down to the minute), I prepare a wet mix and a dry mix the night before (remember, wet mix in the fridge for safety). When I am ready to bake, I just mix the two together, scoop into the muffin tins, and bake.
Best Answer
The main reasons are speed and convenience. Yeast takes longer (even "instant" yeast) and requires more maintenance: waiting for dough to rise, etc.
But those are the historical reasons for the adoption for baking powder. Since it became common, another reason emerged: that is, it's very difficult if not impossible to get some kinds of texture and crumb that you get with baking powder by using yeast.
Why? As dough sits (necessary for yeast to rise), gluten chains begin to form. The longer the dough sits, the more gluten chains you get. Even with pastry or cake flour, the dough will get tougher over time (though obviously not as much as with all-purpose or bread flour).
So, yes, you can make banana bread with yeast. But by the time your dough has risen, it will have become more "bread-like," with a chewier crumb, rather than the tender "crumbly" texture most people are used to in modern quick breads. That is even more true if you de-gas the dough and let it rise a second time, which is generally necessary (1) to get maximum rise, and (2) to even out the gas bubbles so you don't end up with large holes in the final product.
There are entire genres of "yeasted cake" recipes, which used to be quite common in Europe. Even with a lot of sugar, they are often "bread-like," sort of like sweet brioche. They can have a fine crumb, but they won't have the tenderness of a cake made with baking powder (or a quick bread like banana bread, for that matter, which is really like a cake/muffin).
There are entire genres of modern cakes (and quick breads) that only became possible with the use of baking powder (or at least baking soda with an acid). Before chemical leavening, you were limited in tender cakes to a few genres that depended on whipped eggs or egg whites (e.g., angel food cake), creamed butter/sugar (e.g., pound cake), etc.
Aside from texture, there's also a flavor issue. Generally most people like the added flavor from yeast, but it also can add a "bread-like" or "yeasty" aspect to the flavor, which may not be desirable in some cakes. Modern baker's yeast is better than historical yeasts in this regard. Baking powder can add a different (more "chemical") flavor, but with proper proportions and/or with acid in the batter to ensure a complete reaction, it is often barely noticeable.
EDIT: SgtStens makes a great point in another answer, which I was thinking about in my final paragraph with flavor, but didn't note clearly. High sugar content in doughs/batters will inhibit yeast activity. That's why most traditional yeasted cake recipes end up using a relatively low-sugar batter and then add more sweetness through a glaze, filling, streusel topping, etc. A low-sugar dough can still rise within a reasonable amount of time (a couple hours). On the other hand, high-sugar cake batters generally take a much longer time to rise. (Many recipes require an overnight rise for the yeast to initially develop and multiply.) Depending on the type of yeast and temperature, this can sometimes result in an overly "yeasty" flavor to the final product (though the high-sugar content will cover it up partially).
However, the higher sugar amount can perhaps allow you to obtain a crumb and texture closer to a "normal cake." On the other hand, without adequate gluten support (with lower flour content relatively), it's difficult to obtain a very light cake this way. It's fine to create a product with the texture of pound cake or maybe a quick bread, but for something lighter, you may need to resort to baking powder. One reason in this case is that baking powder can continue to react and produce gas until the gluten structure sets, whereas yeast will die a little earlier in the baking process. Very light cakes often depend on a careful balance between continuous gas production inside the dough and the gradual migration of gas bubbling out the top. The cake has to set while gas production is still going on, or else the gas will bubble out and the cake will fall. If the dough rises too high and then the yeast stop producing gas when they die, the cake could collapse before the structure sets.