It depends a lot on the kind of traffic you expect and the condition of the ground. I'm going to write from my experience, outside Seattle, WA, USA, where the subsoil is glacial till.
For light foot traffic, pea gravel can be nice. Rocks are under 1", uniform in size, and have smooth surfaces. Every time you step on it, the rocks will shift, softening the footfall. This is also noisy and causes the gravel to spread out. You can contain it with edging, or just let it spread and refresh it periodically. It's not good for wheeled traffic, as the wheel will just sink in.
If you want to push a heavy wheelbarrow, ride a bicycle, or drive on gravel, you want to pick something with sharp edges. It also helps if the rocks are a mix of sizes. They will lock together over time, providing a sturdy surface.
If the ground is soft and collects water during your rainy season, smaller gravel will just get pushed in to the mud with traffic. An initial layer of larger rock (say, 3" minus) will stabilize things, and a top layer (say 3/4" minus) will give a smooth surface. Mud will still work its way to the surface, but the ground will remain stable. In very wet soft ground, I've heard that a layer of 3" - 5" crushed rock works well under the 3" minus, but I haven't tried it.
To keep mud from working its way up, put a layer of landscape fabric down first.
Look at the shape of the land and the history of draining. Avoid low, wet areas, or reshape them to drain elsewhere. Consider how your traffic will wear and shift the ground, and how that will affect draining in the future, too (e.g. tires leaving ruts).
It a good idea to remove topsoil and other organic matter first. Keep it, though, and use it elsewhere.
Best Answer
You said it.
Start by checking your downspouts to make sure they direct roof water runoff away from your house.