Learn English – welcome baby party vs baby shower party

In this case that is a demonstrative pronoun standing in for the phrase "the results of the calculation". So it is fine as it stands, except for one modification

[The results of the calculation] of number four [are] pretty difficult to get.

Note that the number of the copula changes from singular to plural, is to are, since the antecedent (results) is plural.

Edit:

In the case I cited above, it is the copula that would change. In your case, splitting the thought over two sentences, the demonstrative pronoun that would change number as well:

Here are the results of the calculations. Those of calculation number four are pretty hard to get.

This is only if number four would have "results" instead of "a result"; it's a minor point, but if you mean

The result of calculation number four is pretty difficult to get.

then the singular "That of ... etc." would be fine. Whether or not there is a single result or multiple results for a single calculation is not clear from the example.

Latter is at the end. Being the second of two persons or things mentioned; near or nearer to the end. It has more to do with placement, so to speak. Location, real or figurative.

Between captain and major, the latter is the higher rank.
My favorite is the latter part of the book.

Later has more to do with time.

She arrived an hour later than expected. Better plant your tomatoes later in the Spring than sooner; tomatoes don't do well in the cold.

They are similar, but not interchangeable. You can't say "I'd prefer the later" if asked if you would prefer coffee or tea. You can't say, "Plant them latter in the season."

You can say, in the latter part of Spring, as with the latter chapters of a book.

You can say, I'd like some coffee, but later, please.