Yes, a very dense dough will not be able to rise. The gases will not have the strength to push the hard dough apart. There are types of bagels which are supposed to be quite dense, but American bagels are seldom made this way. This recipe is quite low hydrated at 52%, so even small errors can push it into problematic territory.
Individually packaged instant yeast keeps for a very long time. Big packs keep well if refrigerated after opening. Your yeast is not likely to be the problem. There are three possible problems you may have encountered:
- You measured your dry ingredients by volume, not weight
- You used the wrong type of flour
- The recipe gives an insufficient amount of time
Measuring Ingredients
I suspect that you probably measured by volume. This is the most common mistake which leads to a wrong dough consistency. You should measure by weight because measuring by volume is not precise enough: depending on your technique and on the humidity in your pantry, you can have up to 50% measuring error. The recipe gives you a weight; use it. If you find a recipe which doesn't give you a weight, consider changing the recipe - the best sources always use weight. If you insist on using a recipe specified in volume units, use a converter and measure everything in weight. This ensures that you use the ratio given in the recipe and eliminates technique errors on your part. If the recipe author didn't make technique errors while creating the recipe, you will get the right output. (For example, Corriher calculates her recipes by weight ratio, but she lists them in volume because her readers prefer it that way. If you convert back by the factor she gives, you will never be wrong). If you absolutely must measure by volume, then pour your flour into the measuring cup and level it by shaking. Never scoop flour. Still, prepare for botched doughs from time to time, even with pouring.
Flour Type
The second likely reason is the wrong type of flour. Or maybe not "wrong" per se, but different from what the author had in mind. In different regions, flour labeled as "bread flour" has different amounts of gluten. If in doubt, look at your nutrition label; it should be around 12% for bread flour. If it's not, adjust the amount of water slightly (you have to go by feel here).
Timing
The time given in the recipe is also shady. At 4.5 g dry yeast and 453g flour, it is equivalent to 3% fresh yeast (even less if you overmeasured the flour). This is a good percentage for a slow, long rise. But saying to ferment it in the fridge "for at least one hour" is nonsense. This amount of yeast needs more than an hour at room temperature, or something like 6 hours in the fridge, even for a rich wet dough - probably more for the lean dense one. If you only kept it for an hour before shaping, you didn't have any primary rise. Next time, just go by volume. Primary fermentation should double the dough volume. The change in volume in secondary fermentation depends on the process, and I don't remember the correct one for bagels.
Bottom line: your diagnosis sounds very likely - not enough fermentation for the amount of flour. Measure correctly, and allow enough time for fermentation, to get your bread right.
First, it depends on whether your malt syrup is diastatic or non-diastatic. (Diastatic contains active enzymes; most syrups are non-diastatic because the process to create syrups usually involves heat that destroys the active enzymes. However, this is not universally true.)
Diastatic malt's active enzymes help convert starch into sugar. Some bread or bagel recipes use diastatic malt to speed rising (by converting some starch to sugar quickly to give food to the yeast), as well as to condition the dough and soften the crumb somewhat. The extra rise can also sometimes lead to a lighter result, and the extra sugar can lead to more flavor and good color. One has to be careful about adding too much diastatic malt to a dough, because it can weaken it too much and produce a "gummy" texture. Bagels would likely end up flat and/or with an "undercooked" texture.
Non-diastatic malt is just used for flavor and color. It has no active enzymes, just a malty flavor which is generally more pronounced than diastatic malt products, due to additional roasting and concentration of flavors not possible at lower temperatures when trying to keep enzymes around in diastatic malt.
A little bit of diastatic malt can be useful in bagel dough. It will affect texture as well as giving sweetness and color. Non-diastatic malt (in powder or syrup form) is just about giving sweetness and promoting color. If you're adding malt to the water to boil bagels, there's no point in using diastatic malt, since the boiling will destroy the enzymes. If it's all you have on-hand, however, it can still give a malty flavor and promote color when used in larger quantities.
If both of your options are non-diastatic, there's not a huge difference between powder and syrup. Powder's a bit easier to handle for dough; the syrup is notoriously sticky. But syrup is sometimes easier to dissolve if you're using it in boiling water. The more important difference may be in flavor: you specifically mention "barley malt syrup" vs. generic "malt powder." Some malt powders are made from other grains and may not have the sweetness or complex character of barley malt. The composition of the malt and its specific flavor (which might be developed in different roasting techniques, even if the same grain is used) can have a greater impact than whether one chooses malt powder vs. syrup. I do believe, however, that malt syrups often have a darker roast than non-diastatic powders, so that may be useful for more color (especially in the boiling step). Unless you're buying specialty "dark" malt powder, that is.
Also, the quantity used is most important: in bagels, the contribution of malt to the flavor is usually relatively small (though some would say essential). Unless you're using a lot of malt, you probably won't notice a significant difference between the types of non-diastatic malt in terms of flavor, texture, etc.
Best Answer
Those look like great bagels, as a native New Yorker living abroad in a place with no bagels I both salute and envy your results! I have also been baking my own to get my fix.
Baking bread on a wire rack is generally a bad idea, and a very bad idea with bagels. Bagels are very sticky because you boil them, this gelatinizes the outer layer of the dough, making it very soft so it will mold around any shape. They will sink onto any shape you put them on no matter what you coat them with and then bake onto that shape, so you need to bake them on a flat surface. They will also stick like glue to a flat surface, and there's one or two ways to counteract that:
I usually do both just to make sure.