Learn English – Connotations of “off the reservation”

You asked for a plausible development, so here goes... note that I can't prove that this is how it developed, but I've at least tried to give a genuine usage example of each of the senses in the sequence, with dates where possible.

(i) This phrase was originally used in betting on sports, and in particular horseracing. The meaning is that not only is the bookmaker not taking any more bets, but that existing bets placed on the competition are null and void. This is usually because the race has been cancelled, but can also be due to other factors - for instance, the starting lineup changing so substantially that forcing people to honour their bets would be unfair, or in the event of an irregularity in the conduct of the race. (It can also be because the event was drawn.)

Here is a modern example in horseracing rules:

Bets are off and any stake paid shall be returned to the investor in the following circumstances - [...]

[ ...furthermore... ]

(1) If, in the opinion of the stewards, bets have been or may have been affected by fraud or corrupt practice, they may declare the bets off.

(2) In the event of a race being rerun all bets are off.

Or it can refer to a draw (1897):

An honest judge will be compelled to decide that the race is a drawn one and all bets are off.

Or an irregularity (1901):

...you hired some man to frighten my horse. Of course we either run again or all bets are off.

Or a change in the starting lineup (1839):

Grey Milton is also disqualified for the High Ash [...] Unless coupled with other horses, all bets are off.

Or because of misinformation about the race (1881):

As in the racing world all bets are off when the horse that wins has been erroneously described, ...

I believe that the phrase became embedded in people's consciousness, since at racecourses (and other sporting events) it would commonly be announced "The 14.35 has been cancelled; all bets are off." However, given the oral nature of such announcements, it is hard to track down any evidence, so this is largely speculation, but it is still occasionally used in a manner consistent with this, for example (2009):

The FED needs to announce "All bets are off" concerning the Credit Default Swap 'insurance' scam.

(i)(a) By direct association, the phrase is often used by the media with the implication that the event in question has been cancelled (2008):

All bets are off at Cheltenham as racing postponed due to high winds

Organisers of the Cheltenham Festival have had to cancel the second day of racing after overnight storms

(ii) From the first meaning ("bets are null and void") is derived a sense of all existing agreements are cancelled: that there is a return to the status quo ante, because even existing bets are not honoured (although the stake is returned). Examples of this usage are hard to tell apart from the meaning in the next paragraph, so I have put them all together there.

(iii) Following on from that (and no doubt assisted by the similarity of the phrase the gloves are off), the phrase is applied to situations where the circumstances have changed substantially, to the extent that former truces, alliances and restraint no longer apply.

This can mean that former commitments of non-aggression are discarded (2010):

"[...] if we see states developing biological weapons that we begin to think endanger us or create serious concerns, that he reserves the right to revise this policy [of not using nuclear weapons]."

Clinton added, "If we can prove that a biological attack originated in a country that attacked us, then all bets are off."

Or an amnesty is cancelled (2009):

ATO offers amnesty [until June], then 'all bets are off'

Or former leniency will be reversed (2011):

[after not sending Clarke to prison for a long time] The judge warned Clarke, who has a long criminal record: "The next time we meet, all bets are off and you will go inside, do you understand?"

Or indeed that rules no longer apply (2011):

There is a well-marked area about 30 feet beyond the Start Line, where all bets are off, and there are no speed restrictions.

(iv) Of course, a consequence of this is that, since the rules of the game have changed, there is a sense that "anything can happen" in that what was previously unthinkable is now entirely possible. This sense, probably combined with one aspect of the original meaning (i.e. the circumstances have changed such that former bets/predictions are void), then leads, finally, to the definition quoted in the question - basically the situation is unpredictable and anything could happen.

For our last example, where better to find unpredictability than in the economy (2010):

"The only consensus is that no-one knows what will happen [...] Retailers we've spoken to have all said the same - performance will be slow but steady until summer, then all bets are off until the first budget post the election."

It doesn't come from a derogatory term.

It appears to be a combination of two terms that mean new. Span came from Old Norse and spick came from Dutch.

ORIGIN late 16th cent. (in the sense ‘brand new’): from spick and span new, emphatic extension of dialect span new, from Old Norse spán-nýr, from spánn ‘chip’ + nýr ‘new’; spick influenced by Dutch spiksplinternieuw, literally ‘splinter new.’

New Oxford American Dictionary - Mac Dictionary App

The derogatory term is much more recent.

ORIGIN early 20th cent.: abbreviation of US slang spiggoty, in the same sense, of uncertain origin: perhaps an alteration of speak the in ‘no speak the English.’

New Oxford American Dictionary - Mac Dictionary App