I finally found some time yesterday to take a closer look at the furnace.
Gas Pressure
I started by measuring the incoming gas pressure using a gas pressure gauge. I found it to be 6 WC (inch water column), which is well within the acceptable range specified on the furnace.
Dirty Gap
Next I took a closer look at the spark gap itself, and watched closely as the spark was generated. I even recorded the spark with my video camera, which allowed me to zoom in fairly close. This is when I noticed a bit of carbon buildup on the electrodes, and wondered if it could be that simple...
Cleaning the Gap
I grabbed a bit of 220 grit sandpaper, and cleaned the electrodes up a bit. steel wool is probably the tool of choice for this job, but I didn't happen to have any on hand at the moment (not yet unpacked).
Success!
After testing the pilot ignition a few times, the problem seems to be fixed. We had a cold night last night, and the furnace fired up fine all night. As I write this, the furnace has just cycled successfully yet again.
Update
I found this YouTube video from Grayfurnaceman that describes the problem exactly, and says that the only solution is to replace the assembly. I'll be replacing the pilot assembly once the weather warms up, so we'll see if that solves the problem.
Best Answer
Without knowing the exact make/model of furnace, it's impossible to give a specific answer.
Flame Sensor
It sounds like the furnace may not be detecting the pilot. The furnace uses some type of flame sensor to determine when the pilot is lit. Often times these devices can become covered in soot, and have a difficult time detecting the flame.
Furnaces often have an electronic device that is used to light the pilot, rather than having a pilot constantly burning. One type uses electricity to make the tip of the device heat up enough to ignite the gas, while another uses a spark to ignite the gas. If you have the spark type, that could be the clicking you're hearing. It would be a steady click--click--click. If the furnace is not detecting the pilot, it will continue to send the signal to the ignitor to light the pilot. This may be why you are hearing the constant clicking.
The first step to solving this issue, would be to locate the flame sensor in your furnace. It will be a device with wires leading to it, and it will be in very close proximity to the pilot flame (as in, the tip of it should be in the flame).
Once you've located the sensor, you can clean the tip using fine steel wool (#00, #000). Gently rub the sensor with the steel wool, removing any black soot from the sensor.
WARNING: Make sure the power to the furnace is off; using either the breaker or servicemans switch, before performing any maintenance on the furnace.