Old English
In Old English, a language from the West Germanic family, the standard spelling was tunge (wiktionary) and the corresponding pronunciation was /ˈtʊnɡe/ "tun-ghe" (/ʊ/ as in foot). In other words, the final "e" was not mute but clearly pronounced. For instance in present-day German, you would say "Die Zunge" /ˈtsʊŋə/ (tsung-e). In both cases the final unstressed e is actually already a reduction of an older final a still present in many other cognates.
Middle English
The spelling of Middle English is strongly influenced by that of Old French in which the un sequence is uncommon. It was thenceforth often replaced by on which led to such spellings as tonge or tounge ('ou' being the French spelling of /u/ and /ʊ/) . Which can be compared to contemporary alternative spellings longe and lounge for the word long.
Modern English
With the evolution of the pronunciation of long vowels in Modern English, these spellings became highly misleading. "Tounge" for instance could be read /taʊndʒ/ (as in lounge). As a result, such spellings as tonghe or tongue were preferred. The fact that the simpler spelling tong was not preferred (in spite of the existence of such words as long) is probably a sign that the final e was not completely muted yet.
Finally, the tongue variant probably overtook the tonghe option as a result of an analogy with such words as L. lingua or languages.
Dutch
Also note that on the other side of the English Channel, the Dutch word followed a very similar path. From Old Dutch tunga it successively evolved as Middle Dutch tunge, tonghe and present-day Dutch tong.
Reference
This is the explanation offered by the OED regarding the spelling and the etymology of the word tongue.
[OE. and ME. tunge wk. f. = OFris.
tunge, OS. tunga (MLG., LG. tunge,
MDu. tonghe, Du. tong), OHG. zunga,
zunka (MHG., Ger. zunge), ON. tunga
(Da., Norw. tunge, Sw. tunga), Goth.
tuggô:—OTeut. *tungôn-, held to be
cogn. with L. lingua tongue, for older
*dingua (as lacrima:—dacrima: see tear n.1).
The natural mod.Eng. repr.
of OE. tunge would be tung, as in
lung, rung, sung (and as the word is
actually pronounced); but the ME.
device of writing on for un brought in
the alternative tonge with variants
tounge, townge; app. the effort to
show that the pronunciation was not
(tundʒ(ə) led to the later tounghe,
toungue, tongue, although it is true
that these hardly appeared before
final e was becoming mute, so that its
simple omission would have been
equally effective. The spelling tongue
is thus neither etymological nor
phonetic, and is only in a very small
degree historical.]
I've always assumed that -able/-ible suffix patterns stem from the verbal conjugation of the Latin root, where 1st conjugation usually gives way to -able and the other three to -ible. Of course this goes for the Romance languages, not just English.
Disclaimer: this isn't a rule learned formally, more of an observation-based suspicion that never let me down through countless trials the SAT, my 7th grade spelling bee and that one semester when none of my courses were in English. Oh, Latin, where would I be without you? ;-)
Best Answer
This is an interesting excerpt from Wikipedia:
It reminded me of an article I came across years ago, which explained the disparity between the pronunciation and the spelling of the English language, something not happening in other European languages I know, at least not to this extent. In essence, it claimed it is difficult for written forms of the language to change and follow the spoken changes, especially in English. I don't remember the reasons why, but this explanation was enough for me to understand why there are practically no rules for pronunciation.
As to why these odd spellings haven't been replaced yet, I know there are suggestions for it (not only about English, mind you, a big debate has been going on in France about the simplification of French spelling), but no decisions have been made so far.