How to seal doors and windows to keep air out

Adding that EPDM weather stripping will help if the leak is between the sash and the window sill. I actually had a piece of that stuff in my car to stop a door frame air leak for many years.

I would also check to make sure the air isn't leaking between the sashes of the window as well.

Once you put the weather strip in place, I would go back and double check the perimeter of the window frame for leaks using the old candle test to check for any drafts.

Additionally - Aluminum windows are notorious heat transmitters (they let in lots of heat from outside in summer and cold in winter). I've noticed on cold days aluminum window frames can be so cold relative to inside it almost feels like a slight draft due to convection or whatever effect is in place there.

One more overlooked spot you may want to look at to seal up is your electrical outlets - they are responsible for leaking a lot of air as well. You can get gaskets which go on underneath the cover plates that will seal that leakage up.

Good luck!

You should make sure the door and the window are caulked all the way around the trim.

Sometimes painters neglect to caulk the tops of windows which eventually leads to leaks.