How deep must a range hood be to effectively clear cooking vapor

The size of the hood does matter. As well as the distance from the cooking surface since this impacts the capture of the smoke and odors.

If two hoods have 900 CFMs, but one is 36 inches, and the other 30 inches, you will probably get better exhausting from the smaller hood. Because the velocity of air entering the hood is higher with the smaller hood area.

It won't make much difference if you buy a flat glass or curved glass hood because the shape of the entrance of the air didn't change much if it's still a rectangle.

The CFM to duct size question is mostly a matter of actual CFM, the flow will be reduced with a smaller exhaust duct.

You can use a smoke stick or matches to test the capture affect of your existing exhaust. Start far away from the stove top and move closer until you can "see" the air moving towards the exhaust. You can get an idea of how well your existing exhaust works and what you need to improve the design.

You could try cutting some cardboard and covering the existing opening. Either a flat piece with a rectangular opening or a more elaborate reducer. Then apply the smoke again and see how the capture changes. Put your and at the opening before and after you do this so you have somewhat of an idea of how the air velocity changed with open area.

Never seen one like this - BUT have installed many a hood in cabinets where the vent did not end up exiting the center of the cabinet. a combination of adjustable elbows will offset the vent if you have at least 15" height in the cabinet. As a last resort flexible aluminum duct could be used but only if you have no other options.