When English arrived in the country, it was (makes sense) from the English.
To over-simplify the complex non-native history, and attempt to dodge any political discussion, the English kinda chased the ex-Dutch (Afrikaans) people across the country.
As a result, there became sort of four regions of non-native speakers. The Cape (around Cape Town), Orange Free State (around Bloemfontein), Natal (Around Durban) and Traansvaal (around Johannesburg).
Because of the ports of Port Natal (now Durban) and Cape Town being more English, and the center and north being more Afrikaans, even though most of them speak English now, there are distinctly different accents, even in the same country.
These days, many non-colonist descendents may also speak it as their first language.
As a result, anyone from South Africa where they are speaking it as their first language can generally be considered to speak SAE. Of course in some cases (like myself) as you move to other countries your accent changes, so I now have a SAE/Kiwi/English/start-of-Canadian-English accent forming ;) But when I get a cold, my SAE starts becoming more pronounced...
Simplified summary:
Two main groups who generally spoke English as a 1st language - from Dutch descent (Afrikaners) and from English descent (English). They have similar 'South African' accents - as one has affected the other, but the Afrikaner accent is usually deeper, some might say harsher.
Then there are the locals / natives who were there before the Dutch/English - Zulu, Xhosa, Ndebele etc. They mostly(!!) spoke their own languages, and have English as a second language (there are, however, always exceptions). They too, have different accents, but are still often recognisable to the trained ear as 'South African'.
!! - This is becoming less common after apartheid as there is more integration between the races and communities in social and work circles, and as travel around the country becomes easier and more common.
EDIT:
This link on Dialects and Accents of South Africa may help, as it has a variety of samples of accents explaining the speakers' backgrounds.
Best Answer
You might want to be careful using it with black South Africans, especially those who are culturally or traditionally inclined. "Old-school" black South Africans do not like to be called names they don't understand. In African culture every word has a meaning; colloquial language is considered to be rude especially when being used with elders. Do not say "Saffer" to any South African over the age of 35; that's where the modern boundary is drawn. However, you can ask; you might be surprised.
I am a young modern South African black female.