Before the days of modern tailoring, such garments, whether underwear or outerwear, were indeed made in two parts, one for each leg. The pieces were put on each leg separately and then wrapped and tied or belted at the waist (just like cowboys’ chaps). The plural usage persisted out of habit even after the garments had become physically one piece. However, a shirt was a single piece of cloth, so it was always singular.
It's a matter of the signifier versus the signified. The question being asked is, "why is the word pants plural?" It could as easily have been, "why do we need more than one pant?"
In the first case, since the subject for discussion is a single word, the singular form is correct (if a little odd-sounding at first). In the second, the subject is the article itself, so the option is to pluralize everything or propose a hypothetical singular.
If you write fleece to mean "a jacket or other garment made from a soft warm fabric with a texture similar to sheep's wool," then the plural is fleeces; if you are asking the plural of fleece jacket, that is fleece jackets.
As fleece is also used when talking of a jacket made of fleece (whatever it is sheep's wool, or the soft warm fabric similar to sheep's wool), there is no need to say fleece jacket.
Best Answer
A quick search led me to the excellent site World Wide Words run by Michael Quinion
The site has an entire page on this issue. Here's a brief snippet.