What is "so" when a sentence begins with it?
It's a discourse marker, like oh, well, now, and many others.
It can be used…
To inform the listener that something is relevant to their interest: “So, Sam was asking about you the other day…” suggests something in the air between them, like love, tension, or a bad smell.
To introduce a story, explanation, or change of topic: if I ask someone “What happened on your vacation?” and they begin with “So…”, I’m going to make some popcorn.
To relate a statement to the existing topic, a metaphorical extension of its “therefore” sense to “considering that…” or “in light of what we’ve been discussing…”.
As a generic discourse marker, to take a moment to gather one’s thoughts, just like the others above.
When did it start?
It’s hard to say. (I can’t find a satisfactory source.) Formal discourse markers, such as Beowulf’s hwæt mentioned in another answer, are well attested in written records. They particularly appear in texts of the sort meant to be performed—poems, epics, songs, plays, and so on—that is, not so much in tax records and epitaphs.
So in particular has been used in roughly this way, meaning “thus”/“therefore”, for hundreds of years, since early modern English. NPR claims that the specific use of so as a discourse marker for introducing an explanation rose to mainstream infamy from about the 1980s to the early 2000s, possibly influenced by the English of Silicon Valley during the tech boom.
Is it just a "pause" word…
No, but, so, it can definitely function that way too, so, yeah.
…(and is there a word for that)?
There is! That’s a filler word, such as “um”, “like”, “er”, “ah”, and all those other little interjections. “Just” is a bit belittling: filler words serve an important pragmatic role in conversation, namely, they signal that you’re thinking or pausing but still holding the floor.
As it happens, I rarely use those kinds of filler, and as a result, I’m often met with “What?” or interrupted, because I pause without an indication that I’m not done yet, so people tend to assume I am done. They either have trouble parsing my half-sentence as a full one, or mistakenly think it’s their turn. I’m not quite following the rules for discourse dynamics, and it trips us up.
Is it grammatically correct?
Yes. It’s used and understood by native speakers of many dialects now and I’d say it forms an everyday part of informal spoken standard English.
Am I the only one that finds it annoying?
Nope, in the late aughts to early teens (when this question was originally asked) there was a spike in grumbling about this usage as it reached mainstream saturation. As always, you’re free to get annoyed by any language change you like, or rather, dislike, but it’s worth reminding ourselves that “I don’t like it” is not the same thing as “It’s bad”.
Best Answer
The Grammar Girl has a good article on this topic, basically:
If it means "to whatever extent", don't use a comma:
If it means "nevertheless", use a comma: