From the limited experience I have of hearing English spoken by South Africans or even knowing someone is South African (from real life or movies), I find it impossible to really tell when someone is speaking with the accent called South African English (SAE).
I've done some looking up at wikipedia: South African English, a summary of languages there and more in depth, but it's hard to get a real perspective. The latter says:
"Despite the fact that English is recognised as the language of commerce and science, it was spoken by only 8.2% of South Africans at home in 2001"
I expected (as is borne out by that page), that the great majority speak a native African language (Zulu, Xhosa, Sotho, Tswana, etc.) but I expected a bit more parity (not knowing history well otherwise) between the two colonial languages. But Afrikaans has only 13% speakers, and English only 8% (percentages are for first at-home language not including 2nd languages).
English, the country's lingua franca, is the language of business, politics, and modern communication media, but ranks sixth in home usage.
With that context in mind, what is the SAE accent? Who speaks with that accent? Presumably the descendents of English colonists speak that.
Rather the question I have is, do Afrikaners speak SAE accentless, or is there an Afrikaner accent in SAE, different from a Brit speaking SAE?
That is, did Pieter Botha or F. W. de Klerk the guy in the District 9 movie or Matt Damon in Invictus, did all these people speak SAE with the recognized average SAE accent, or did they speak it with an Afrikaner accent (since they are all Afrikaners at least by name)? My expectation is that Afrikaners and native Africans speak with English with something slightly different than an Anglo SAE accent.
Best Answer
When English arrived in the country, it was (makes sense) from the English.
To over-simplify the complex non-native history, and attempt to dodge any political discussion, the English kinda chased the ex-Dutch (Afrikaans) people across the country.
As a result, there became sort of four regions of non-native speakers. The Cape (around Cape Town), Orange Free State (around Bloemfontein), Natal (Around Durban) and Traansvaal (around Johannesburg).
Because of the ports of Port Natal (now Durban) and Cape Town being more English, and the center and north being more Afrikaans, even though most of them speak English now, there are distinctly different accents, even in the same country.
These days, many non-colonist descendents may also speak it as their first language.
As a result, anyone from South Africa where they are speaking it as their first language can generally be considered to speak SAE. Of course in some cases (like myself) as you move to other countries your accent changes, so I now have a SAE/Kiwi/English/start-of-Canadian-English accent forming ;) But when I get a cold, my SAE starts becoming more pronounced...
Simplified summary:
Two main groups who generally spoke English as a 1st language - from Dutch descent (Afrikaners) and from English descent (English). They have similar 'South African' accents - as one has affected the other, but the Afrikaner accent is usually deeper, some might say harsher.
Then there are the locals / natives who were there before the Dutch/English - Zulu, Xhosa, Ndebele etc. They mostly(!!) spoke their own languages, and have English as a second language (there are, however, always exceptions). They too, have different accents, but are still often recognisable to the trained ear as 'South African'.
!! - This is becoming less common after apartheid as there is more integration between the races and communities in social and work circles, and as travel around the country becomes easier and more common.
EDIT:
This link on Dialects and Accents of South Africa may help, as it has a variety of samples of accents explaining the speakers' backgrounds.