No, you cannot. Sadly, the achievement only registers on the boss kills, and not by going further down the dungeon.
After several careful playthroughs taking notes, I believe I can answer my own question.
Enemy waves are indeed set, but this is only 95% accurate. The enemy waves for each zone (Basement, Caves, The Depths, etc.) are always the same (two for each zone, other than basement, where I've only ever seen a challenge room on the second floor and Sheol, which only has one floor). However, there is no set association between floor and wave, meaning that if we have floors 1 and 2 and waves A and B, the association of floors to waves could be either 1-A, 2-B or 1-B, 2-A (though this loose association might be new in light of the Halloween update, as prior to this I've have always had the same wave on the same floor in the same zone). So why 95%? It's almost always 1-A and 2-B from my experience (even after the update).
So here are the waves for each zone (check the wiki for monster and boss names):
- Basement:
- Wave A:
- 5 x Fly (the blinking red ones, not the docile black ones)
- 5 x Pooter
- 5 x Gaper (pretty sure either kind can spawn, usually the ones with shut eyes)
- Caves:
- Wave A:
- 5 x Charger
- 4 x Globin (usually the slow kind, but fast ones can spawn)
- 4 x Maw (either kind, but usually 3 white and 1 red)
- Wave B:
- 5 x Spity
- 3 x Hive
- 3 x Host (both kinds, but usually it's 3 armored ones)
- The Depths:
- Wave A:
- 4 x Baby
- 5 x Hopper
- 4 x Knight (the faster version new to the Halloween update can also spawn)
- Wave B:
- 3 x Vis
- 4 x Leaper
- 4 x Globin (always the fast ones, as far as I have seen)
- The Womb:
- Wave A:
- 3 x Larry Jr. (each just a head and one segment)
- 1 x Chub
- 2 x Monstro
- Wave B:
- 1 x Fistula
- 1 x Loki
- 1 x Chub
- Sheol
Seeing how Sheol is rather new and also a pain to walk around needlessly, I've only been to the challenge room one time. I got The Womb's wave A, and if I had to guess you simply get one of The Womb's waves. If I find out otherwise I'll be sure to update this answer (and feel free to shed some light on this point yourselves).
Best Answer
Under most circumstances, no. However, there may be a way to get around that. If you get curse of the labyrinth for your final floors, after you beat the first of the two bosses, it will glitch and give you the achievement and the unlockable. Theoretically at that point you could use the shovel to go to the next floor and continue playing. I can not actually verify that, but it is very likely, although in practice incredibly difficult to do. You are better off trying to beat challenge 1 and then using the candle to easily beat mom (it goes through closed doors) and the devil in sheol (it can hurt the devil while you are fighting the fallen). Hope this helps a bit.