Alright, first off a quick summary of the two verb tenses you're asking about.
Past Perfect [had verbed] - This indicates that something happened further in the past than the time I'm referring to. If I was asked "Was that night the first time you drove a car?" I would answer, "No, I had driven once before that." The event I'm referring to (my first time driving) was already in the past when the time already in question (that night) occurred. Past perfect is the correct verb tense in this context.
Past [verbed] - The simple past tense just indicates that something happened before the present. It is used when I am referring to a single event in the past or to multiple concurrent events. For example, "I went to Europe" or "I was in Belgium when they won the World Cup."
I do agree with StoneyB's comment that the question/response you've listed sort of lacks a logical relationship without a story around it. I also agree with FumbleFingers' comment that (again, without context) being out of town "until that moment" seems odd. So, the correct verb tense really depends on what you're trying to say. Presumably it's that you were not in town when this event happened last time. In that case...
How many events in the past are we referring to? Just one. The "last time" this thing happened, during which you happened to be out of town. Therefore the simple past tense is the correct one to use.
"It couldn't have happened, because I was out of town [at] that moment."
If my assumptions to what you were trying to say are incorrect please just edit the question or post a comment to explain that and I'll be happy to edit my answer.
Let's compare your examples:
The party was great, couldn't have been better!
This expresses an extreme. Since the party couldn't have been better, it's as good as it could possibly be. That makes this a fairly strong statement! It's even stronger than the party was great, so it works as a bit of superlative emphasis.
The cake was too small. It could have been bigger!
This doesn't express an extreme. The range of possibilities is anywhere from a little bigger to a lot bigger. As a result, this statement is fairly weak. It doesn't strengthen the statement made by the cake was too small at all. As it is, the second half of the sentence should be removed.
If you want to make a strong statement about the cake using this construction, you need to come up with a negative extreme that you want to express. (It doesn't have to be literally true; it can be hyperbole.) For example:
The cake was too small. It couldn't have been smaller!
This is a little better, though it sounds a little silly. (It's not the world's finest example of hyperbole.) You could make a more general statement about its quality:
The cake was too small. It couldn't have been worse!
This is a bit less silly.
By the way, I'd like to recommend Daniel's answer, in particular for his description of "could have been better", a phrase which is intentionally weak.
Best Answer
You couldn't have got used to is fine, but in US speech it would be gotten, and I think in conversation you'll more likely hear You couldn't have gotten too used to that. We try to keep modifiers as close as possible to the terms they modify.
Assuming that they're co-referent, I'd flip the that and the it, or make both pronouns it. The that points to something outside the sentence, so it should come first; it then establishes a convenient referent for it.
The perfect construction in the first clause is unnecessary as it stands—see here and other questions under the past perfect tag. Simple past will do:
But if you flipped the two time references it the perfect would be just dandy: