Building names are proper nouns. The given (or official) name of the structure you mentioned is "American Radiator Building"*. Thus, "Building" should always be capitalized in this name. On the other hand, the is not part of the given name. So, in this sentence:
I visited the American Radiator Building yesterday
it would be incorrect to capitalize the thus:
I visited The American Radiator Building yesterday
When writing an address, the official name of the building should always be used. It follows that all the words in that name should be capitalized. Your example,
7 World Trade Center
is correct. One might also see/hear "World Trade Center" used within a sentence, for instance:
Memories of the tragedy at the World Trade Center remain fresh in our hearts.
It would be wrong to capitalize the in the above example, based on the aforementioned reason.
*Interesting note: Though still widely referred to by its old name, "American Radiator Building" was officially renamed "American Standard Building".
Best Answer
In its chapter on numbers (and select other sections), the Chicago Manual of Style has these recommendations: