What masonry anchor should I use to install both window boxes and shutters into brick

No, this isn't normal. Your anchors don't fit to your screws. What is happening is that the tip of the screw reached the concrete, and, of course, you can't force a screw into concrete - not even with a drill or a electric screwdriver, and certainly not by hand.

This picture shows how anchor and screw should fit into the hole. The hole definitely needs to be longer than the screw. The screw should be a bit longer than the anchor, but not as long. Maybe the clerk saw that you have picked an wood screw and gave you anchors suited for the second situation. It doesn't hurt to use wood screws with anchors in a concrete wall, you'll just have the cap jutting out a bit. But the length of the screw should be only a bit longer than that of the anchor (the surplus should equal the length of the tip + the length of the attached thing), and your cap will stick out a bit if the thing you are attaching isn't soft enough for the cap to sink in.

The table shows how to determine the correct hole depth. It also tells you what diameter of anchor and drill to choose for a given screw. As for the length of the screw, it is determined by the weight it will have to carry. Vertical screws are more problematic than screws in walls, because gravity is pulling in the direction of the hole, not at 90° to it. Light curtains will be OK with your size screw, but don't put any molton on them.

If you want to use a screw I would use Tapcon® Concrete Screws. They hold very well and are fairly easy to use. The only draw back is that they are a bright blue and you will have to countersink them and caulk or plugged the hole. 3/16 inch will be strong enough, you will want about 1.5" of penetration. To install you just drill a hole matching the size of the screw (needs to be exact size required) and screw them in. Using a screw is helpful because its about the only concrete anchor you can easily remove.

The other option is Metal Hit Anchors. These are a nail in a sleeve. you pre drill the hole and hammer the anchor in and the nail expands the sleeve.

Any time you drill into masonry you should be using a hammer drill, and a masonry bit that is specifically designed to be used with a hammer drill.

On most brick homes anchoring into the brick is going to be stronger. But it is only shutters and you could get away with drilling into the mortar. This way if you ever want to take the shutters down you can fix the hole in the mortar, unlike if you were to drill directly into the brick.