Perhaps more important than the size of the screws (or other fasteners) is what they actually grasp. With any significant load, a screw will not hold by itself in tile or plaster. The tile will chip and the plaster will crumble.
One possibility is to use an anchor through the tile and plaster that expands laterally. There are several types (plastic, metal, etc.), but these are also generally used for lighter loads that are close to the wall, not a shelf that sits on brackets that act like levers, tending to pull them through the wall:
![plastic anchor](https://i.stack.imgur.com/A6Iu0.jpg)
If your plaster is attached directly to masonry, there is an expanding anchor that is meant for that:
![lag anchor](https://i.stack.imgur.com/OvqJW.jpg)
It must be placed in the masonry, not the plaster.
If there is a hollow space behind the plaster, a third type of anchor that mushrooms behind the surface could be used:
![mushroom anchor](https://i.stack.imgur.com/eE9Oq.jpg)
A fourth kind is a toggle that is inserted folded, but then opens behind the surface of the plaster and its fastening:
![toggle1](https://i.stack.imgur.com/1ED3J.jpg)
![toggle2](https://i.stack.imgur.com/i11Kk.jpg)
If the wall behind the plaster is hollow, this may be the best type of fastener.
If there are studs behind the plaster, the strongest fastening is to align the brackets with the studs and go through both the backsplash, plaster and finally into the studs with wood screws (wooden studs) or screws designed for metal studs.
Putting tile over a painted surface can be risky. Sure looks like removing all that paint would be a chore and maybe hazardous (lead based paint if pre 1978). We have put tile over such surfaces by roughing it a bit and putting a scratch coat of "floor patch and leveler" over the entire surface. This is a good bonding agent on stable subfloors.
Another very good alternative is to use one of the new no-mortar interlocking tile products. These are tiles mounted on an interlocking plastic base that just snap together. They are very quick and easy to install. There are several different brands, tile material and backer types available. There is an article about these type of tiles, on Good Housekeeping.
Google "no mortar tile". Lots of articles on the subject
Best Answer
Since you will be applying thinset and tile, you have a lot more "play" in your screw head depth. While you of course want them to be as flush as possible, the tile over thinset will even things out as you apply the tile and make them flush.
If this were drywall, the story would be different.