Why does whole leaf tea leave a residue in the cup

The differences in quality between teas is usually down to the manufacturer and the product line. As far as I can tell, there is no discernible difference between Twining's English Breakfast tea (for instance) in bags or loose leaf. They are the same leaves, presented differently.

However, and this is the important part, there is a difference in how they infuse. Because of the limitations of the tea bag, and the flow of the water through the leaves, a loose-leaf tea is usually infused more quickly, and possibly better. That is, you get more essence of tea from the leaves into the water. There are different bag designs (pyramid bags for example) which try to bridge some of this gap.

The advantage of the tea bags is that they are easier to work with, as you don't need to strain the leaves out of the water, you can just pick the bag up with your spoon.

There is a lovely essay by George Orwell about making "proper" tea. Some of it is a matter of taste, but his point about the infusion is spot on, in my opinion.

TL;DR = The sweet tea takes longer to cool down because there is a lot more stuff in it to get cold.

When cooling unsweetened tea, you are cooling almost pure water (the tea solids are negligible). A 12 ounce glass of unsweetened tea has about, you guessed it, 12 oz (by weight) of liquid to cool, or 340 grams (mass).

Southern-style sweet tea (if this is the "sweet tea" you are referring to), has a 2:1 ratio (by volume) of tea to sugar. A 12 ounce glass of sweet tea has 12 oz (by weight) of tea, and 5-6 ounces of sugar (by weight - 8 oz of sugar by volume = ~7 oz by weight). This puts the total weight of the beverage at 18 ounces or 510 grams (by mass).

The sweet tea, in this example, has 50% MORE mass than the unsweetened tea! This extra mass will take more time to cool down, because there is a lot more STUFF to get cold. It occupies the same volume, but there are a LOT more molecules to chill.

Compounding factors:

• Heat Capacity vs. Specific Heat: A detail to this is that a solution of sugar and water has a lower specific heat (by MASS unit) than pure water, so the total heat capacity of the sweet tea is not quite 150% of the heat capacity of the unsweetened tea, but somewhere between 100% and 150%. Intuition would put it somewhere in the 130%-140% range. Read the physics.SE question linked above for some details on that calculation. Rest assured, however, that the heat capacity of the sucrose solution is higher than pure water.
• Conductivity: I've ignored the thermal conductivity of the solution, since I'm assuming that the stirring in the tea makes the small conductivity differences between the solutions negligible, but that calculation could be done as well.
• Convection: Unstirred sweet tea will experience less convection than unstirred unsweetened tea. In the unstirred sweet tea, dense sucrose solution will remain at the bottom while the cold water from the melting ice will sit on the surface (you can actually see this visually if the tea is sufficiently colored). This slows cooling by slowing the mixing of the cold liquid with the warm liquid. In unsweetened tea the cool liquid will sink the bottom, promoting convection and self-mixing. However, convection is a side-issue to the primary point, the total heat capacity of the beverage.

And as a final note, explanations like this really make it obvious how annoying it is that US measurements use ounces for both volume AND weight.