Alright, first off a quick summary of the two verb tenses you're asking about.
Past Perfect [had verbed] - This indicates that something happened further in the past than the time I'm referring to. If I was asked "Was that night the first time you drove a car?" I would answer, "No, I had driven once before that." The event I'm referring to (my first time driving) was already in the past when the time already in question (that night) occurred. Past perfect is the correct verb tense in this context.
Past [verbed] - The simple past tense just indicates that something happened before the present. It is used when I am referring to a single event in the past or to multiple concurrent events. For example, "I went to Europe" or "I was in Belgium when they won the World Cup."
I do agree with StoneyB's comment that the question/response you've listed sort of lacks a logical relationship without a story around it. I also agree with FumbleFingers' comment that (again, without context) being out of town "until that moment" seems odd. So, the correct verb tense really depends on what you're trying to say. Presumably it's that you were not in town when this event happened last time. In that case...
How many events in the past are we referring to? Just one. The "last time" this thing happened, during which you happened to be out of town. Therefore the simple past tense is the correct one to use.
"It couldn't have happened, because I was out of town [at] that moment."
If my assumptions to what you were trying to say are incorrect please just edit the question or post a comment to explain that and I'll be happy to edit my answer.
Let's compare your examples:
The party was great, couldn't have been better!
This expresses an extreme. Since the party couldn't have been better, it's as good as it could possibly be. That makes this a fairly strong statement! It's even stronger than the party was great, so it works as a bit of superlative emphasis.
The cake was too small. It could have been bigger!
This doesn't express an extreme. The range of possibilities is anywhere from a little bigger to a lot bigger. As a result, this statement is fairly weak. It doesn't strengthen the statement made by the cake was too small at all. As it is, the second half of the sentence should be removed.
If you want to make a strong statement about the cake using this construction, you need to come up with a negative extreme that you want to express. (It doesn't have to be literally true; it can be hyperbole.) For example:
The cake was too small. It couldn't have been smaller!
This is a little better, though it sounds a little silly. (It's not the world's finest example of hyperbole.) You could make a more general statement about its quality:
The cake was too small. It couldn't have been worse!
This is a bit less silly.
By the way, I'd like to recommend Daniel's answer, in particular for his description of "could have been better", a phrase which is intentionally weak.
Best Answer
(I think I'm going to open a can of worms with this answer but I've done some research so, don't blame me.)
In the student's text book, New English File Upper-Intermediate Oxford University press Page 138 it says:
So for some it is considered standard English to use: can't have or couldn't have instead of mustn't have when you are speculating or guessing about the past in questions and negative sentences.
He couldn't have been hungry
means practically the same as
He can't have been hungry
They both express a strong conviction in the past, the speaker can choose to add further information in order to back up his claim.
Thus the speaker is saying it's impossible that John was hungry because he knows John never leaves home without eating something. Must not (mustn't) means something quite different, you are forbidding someone or something from performing an action now, in the present and it is not used for speculating in the past.
On p394 in Practical English Usage by Michael Swan:
This is also confirmed by A Practical English Grammar by A.J.Thomson A.V. Martinet 4th edition on page 148.