You asked for a plausible development, so here goes... note that I can't prove that this is how it developed, but I've at least tried to give a genuine usage example of each of the senses in the sequence, with dates where possible.
(i) This phrase was originally used in betting on sports, and in particular horseracing. The meaning is that not only is the bookmaker not taking any more bets, but that existing bets placed on the competition are null and void. This is usually because the race has been cancelled, but can also be due to other factors - for instance, the starting lineup changing so substantially that forcing people to honour their bets would be unfair, or in the event of an irregularity in the conduct of the race. (It can also be because the event was drawn.)
Here is a modern example in horseracing rules:
Bets are off and any stake paid shall be returned to the investor in the following circumstances - [...]
[ ...furthermore... ]
(1) If, in the opinion of the stewards, bets have been or may have been affected by fraud or corrupt practice, they may declare the bets off.
(2) In the event of a race being rerun all bets are off.
Or it can refer to a draw (1897):
An honest judge will be compelled to decide that the race is a drawn one and all bets are off.
Or an irregularity (1901):
...you hired some man to frighten my horse. Of course we either run again or all bets are off.
Or a change in the starting lineup (1839):
Grey Milton is also disqualified for the High Ash [...] Unless coupled with other horses, all bets are off.
Or because of misinformation about the race (1881):
As in the racing world all bets are off when the horse that wins has been erroneously described, ...
I believe that the phrase became embedded in people's consciousness, since at racecourses (and other sporting events) it would commonly be announced "The 14.35 has been cancelled; all bets are off." However, given the oral nature of such announcements, it is hard to track down any evidence, so this is largely speculation, but it is still occasionally used in a manner consistent with this, for example (2009):
The FED needs to announce "All bets are off" concerning the Credit Default Swap 'insurance' scam.
(i)(a) By direct association, the phrase is often used by the media with the implication that the event in question has been cancelled (2008):
All bets are off at Cheltenham as racing postponed due to high winds
Organisers of the Cheltenham Festival have had to cancel the second day of racing after overnight storms
(ii) From the first meaning ("bets are null and void") is derived a sense of all existing agreements are cancelled: that there is a return to the status quo ante, because even existing bets are not honoured (although the stake is returned). Examples of this usage are hard to tell apart from the meaning in the next paragraph, so I have put them all together there.
(iii) Following on from that (and no doubt assisted by the similarity of the phrase the gloves are off), the phrase is applied to situations where the circumstances have changed substantially, to the extent that former truces, alliances and restraint no longer apply.
This can mean that former commitments of non-aggression are discarded (2010):
"[...] if we see states developing biological weapons that we begin to think endanger us or create serious concerns, that he reserves the right to revise this policy [of not using nuclear weapons]."
Clinton added, "If we can prove that a biological attack originated in a country that attacked us, then all bets are off."
Or an amnesty is cancelled (2009):
ATO offers amnesty [until June], then 'all bets are off'
Or former leniency will be reversed (2011):
[after not sending Clarke to prison for a long time] The judge warned Clarke, who has a long criminal record: "The next time we meet, all bets are off and you will go inside, do you understand?"
Or indeed that rules no longer apply (2011):
There is a well-marked area about 30 feet beyond the Start Line, where all bets are off, and there are no speed restrictions.
(iv) Of course, a consequence of this is that, since the rules of the game have changed, there is a sense that "anything can happen" in that what was previously unthinkable is now entirely possible. This sense, probably combined with one aspect of the original meaning (i.e. the circumstances have changed such that former bets/predictions are void), then leads, finally, to the definition quoted in the question - basically the situation is unpredictable and anything could happen.
For our last example, where better to find unpredictability than in the economy (2010):
"The only consensus is that no-one knows what will happen [...] Retailers we've spoken to have all said the same - performance will be slow but steady until summer, then all bets are off until the first budget post the election."
It's a bit of a pun. While the phrase let them up easy may be rare, the phrase let him down easy is not.
Let him down easy could refer to a man who is fainting – catch him, and lay him gently on the ground – but it could also refer to an emotional blow. For example, one might advise a woman who intends on breaking up with her boyfriend, a soldier who has been deployed for six months, “Don't just break up with him the moment he gets of the plane – let him down easy.”
At the end of the Civil War, many in the Union were still bitter toward their enemies. Lincoln, however, was a proponent of reconstruction; he felt that the Southern states needed to be gradually welcomed back into the Union in a non-punitive manner.
I've not heard the quote before, but it sounds Lincolnesque. Lincoln was rather fond of quips and puns; this one is attributed to him:
The man who can't make mistakes, can't make anything.
Best Answer
As you note the phrase "on the table" (in all uses, including both idiomatic ones and literal ones) is far more common that the phrase "off the table" (likewise in all senses). Here is the Ngram chart for the period 1700–2005 for "on the table" (blue line) and "off the table" (red line):
Used idiomatically, "on the table" goes back several centuries. Christime Ammer, The American Heritage Dictionary of Idioms, second edition (2013) has this entry for the phrase:
Evidently, "off the table" when used idiomatically, indicates the negation of "on the table" definition 1 above. In any event, Ammer doesn't have an entry for "off the table." I suspect that the reason she omits this phrase is that it is a much more recent and much less common idiom that "on the table."
Although Google Books matches for "off the table" do indeed go quite far back in time, the idiomatic use of of "off the table" is surprisingly recent. Moreover, the earliest instances that do occur are concentrated in U.S. government lingo. In Google Books search results for "off the table" for the period 1942–1996, all seven matches for the phrase in its idiomatic sense are from U.S. government reports, and only two of those are from farther back than 1992. The earliest match is from Public Papers of the Presidents of the United States: Administration of Ronald Reagan (March 21, 1985):
And the next-earliest is from "Impact of the President's 1987 Budget: Hearings Before the [House] Committee on the Budget," (1986) [combined snippets]:
The five matches from the 1990s appear in similar contexts. It seems clear to me that the idiomatic use of "off the table" first came to public notice as government slang in the 1980s. In the past decade, I believe, it has become much more common; but it may still be viewed primarily as bureaucratic, legislative, and diplomatic jargon.
In any event, as the 2013 edition of Ammer's book suggests, the reason you can't find an idiomatic entry for "off the table" in reference works probably isn't that the books you consulted are too old, but that very few such books cover the phrase even today.