Learn English – Difference between “I did say that” and “I said that”

that in this pattern refers back to the context supplied by the statement in the prior clause.

I did not eat the cake. It was not that tasty.

that tasty = tasty enough that I would want to eat it.

I finished the cake in just two bites. The piece was not that large.

that large = so large that it would have required more than two bites

I had hoped to see the game. But admission cost $250. I am not that much of a fan.

that much of a fan = so fanatical that I would be willing to spend $250 to watch the team play.

I tried her cake. It was not that delicious.

that delicious = as delicious as I expected it to be | as delicious as other people said it was | so delicious that I will praise it

I love him despite that he is small.

"Despite" can't normally take a that-clause. Most speakers would find this sentence either ungrammatical or unidiomatic.

I love him despite the fact he is small.

This is correct and idiomatic. Some stylists dislike this form (it's rather wordy), but it's quite common among native speakers.

I love him despite him being small.

I love him despite his being small.

These forms are correct too. The former is slightly more informal, the latter (with "his") slightly more formal, though some speakers use only one version regardless, while others vary their usage.

I love him even though he is small.

This is yet another way of saying the same thing (often recommended as a way of avoiding the wordiness of "despite the fact that").