How long should I let latex paint dry before installing outlet covers and other fixtures

I would wait for a couple of days to see if the filler (which maybe absorbed water from the paint) dries completely and then the color matches. But since you didn't prime the filler (or maybe it didn't dry completely), possibly where it was thick, it has modified the color of the paint. If the area is still darker after a couple of days of letting the paint/filler dry, then I would apply primer to those areas and repaint. Hopefully you can use the same roller nap and get a good match on texture so you will not have to repaint the entire wall.

This E-How article says that a soak with your normal shampoo, followed by running a fine-tooth comb through the affected area(s), should remove water-based paints including latex acrylic. By fine-tooth, I assume they mean a comb like a pocket or dressing table comb, but you may have more luck with something like a lice comb.

Now, latex paint with VOCs may cause other types of discoloration, similar to thinners (which is why you NEVER put paint thinner on your hair; no turpentine, no acetone, no methylated spirits, no nothing). Also, certain dyes, like red and orange, will soak into surfaces (even some things you wouldn't consider "porous", like ceramic tile) and become extremely difficult to remove. I dunno what color you painted the closet or exactly what type of paint you used, but it may not be a layer of actual paint causing the discoloration anymore.