Punctuation – Should There Be a Space After ‘No.’ and ‘p.’

For the punctuation marks for which you are asking the question (comma, colon, question mark, and exclamation point), they are written without any space before them, and with a space after them.

This is what you need: canned tuna fish, tomatoes, beans, olive oil, onions, parsley, and garlic.
I cannot believe it! You are accusing me of something you did!
What time is it? I am hungry.

The same is true for the semicolon and the period.

As for what's wrong with the wrong way, it is just not how punctuation marks are used nowadays. If you put a space before the exclamation point, I would think you are French, and that you are writing in English using the punctuation marks as you would in French, since in French you normally write a space before the exclamation point or the question mark.

According to The Chicago Manual of Style^†, section 14.21, this is the normal way to do it:

A note number should generally be placed at the end of a sentence or at the end of a clause. The number normally follows a quotation (whether it is run into the text or set as an extract). Relative to other punctuation, the number follows any punctuation mark except for the dash, which it precedes. (emphasis added)

In my experience (which is mostly from reading linguistics papers), this is the usual practice. I decided to check my memory, so I opened ten papers at random on my hard drive by different authors. I found that nine followed this practice, while the remaining paper placed the superscript reference number before any punctuation. So it seems (at least from a sample of linguistics papers) that this practice isn't followed universally, but I'd nonetheless recommend you follow common practice both for aesthetic reasons and to avoid distracting the reader.

^† Thanks to Jason Patterson for pointing this out in the comments section.