I had to draw a small diagram to see how the area was around the house. I drew a swale in to show what is there already, or hopefully something that can be created
What looks troubling here is the concentration of runoff heading to the house IF there is no swale there, that is why I hope there is one there, if not you really need to get the flow away form the house. It is bad enough that it is sandy, the water can still perk into the foundation, though it will be minimized with a swale. The clay base on the north side, sounds exactly as you describe, a bowl.
You need to do whatever you can to minimize the water infiltration, to start. It may not require to pipe the downspouts away from the house to a drywell or into a French drain, well not a French drain as you may plan around the house. French drains return the water to the ground, and in turn the crawlspace. What I mean is terminating the downspouts into an in ground pipe that daylights downhill of the home. A drywell would work too, but it is much more than may be needed.
This is the minimum I would suggest, although it is still a lot to do. I would also consider waterproofing the foundation, it may work on most of the foundation, but I think the runoff would still migrate under the foundation on the south side since it is still on sand/ clay mix presumably.
All of this is still tempered by the symptoms created by the water in the crawlspace. If there is no evidence of black marks (mildew) or white fuzzy stuff growing in places from the excess moisture in the crawlspace, if none of these symptoms exist, al least extreme measures are not needed. Regrade if needed, would be the minimum, downspouts to daylight away from the house the next active remedy, drywell addition the next, and if you are doing all that digging, waterproof the foundation, all these measures combined are the extreme I mean.
Edit 1/26/13
Aside from the shape of the house and other extensions from it, it shows the same thing happening except your swale that exists is more of a collection point. It appears the rain runoff takes the same path, but the house is in the way. The swale in the above sketch I posted would divert the rain water before it gets too close to your home. Is this a possibility to happen? It can be done by lowering the grade or if the house is high enough above grade, the grade can be raised, but that has implications too. The water in the swale may be of no concern. I would draw more concern to the south and west side where the rain looks like it is path is unencumbered to the foundation.
I will edit your drawing to show a more accurate idea and post it.
Back at the downspouts that I mentioned earlier, if the grade looks like it can daylight a underground lead would help reduce the amount of water at the foundation. Yet the addition of the swale will redirect the downspout water as well.
Given the willingness to devote time and energy to the project, sure you can regrade it, but looking at the photographs it may not solve the root problem.
It appears that water will sheet flow down the driveway and directly into the building...that's why the drain is there. The presence of the french drain suggests that the problem predates the gravel extension.
Diverting the sheet flow well uphill from the building will reduce the amount of water hitting the building. This means regrading a portion of the driveway (perhaps prior to the asphalt to concrete driveway transition would be easier).
Best Answer
You can certainly fill with cold patch, though for good adhesion you would need to start with getting the area really dry, which will be considerable work as it's a low spot that holds a puddle. What may happen, and is more likely if you don't get it well-dried out before applying the cold patch, would be water getting or staying between the patch and the old asphalt, causing the patch to pop loose when that water freezes.
Both the appearance and the odds of having it pop loose could be helped by sealing the whole area after patching to fix the grade, thus giving a more uniform color and also reducing water ingress.
There's no argument that ripping up and replacing with new hot asphalt or concrete would be less likely to fail long-term, and probably look better, but there's also no argument that it's going to cost a lot more and / or get out of your DIY comfort zone, depending where your comfort zone is.
A third alternative (since you are getting out the cold patch anyway) would be to cut a channel from the lowest part of the low spot to the sump, place a drain pipe in it, and then cold-patch the channel.