Learn English – rule for which suffix to use when creating adjectives from nouns

What you are asking about is morphological productivity of English suffixes. Productivity in this sense means using morphemes to coin new words. (It just so happens my thesis deals with morphological productivity!)

There are a number of different aspects of the language system that influence productivity and the makeup of the lexicon.

Synchronic change

As a language evolves through time, different suffixes rise in productivity and later fall in productivity. For example (from Anshen & Aronoff 1999), the suffix -ment entered into the language around 1250 according to the OED, through whole-word borrowings from French (many, if not most, suffixes enter a language this way). It converts verbs to nouns. The number of new words coined with -ment steadily rose up until around the 16th century, when it sharply began to decline. The OED cites only 2 new words since 1950 that were coined using -ment. However, once a word has been derived, and has stabilized in the lexicon, it becomes fossilized. That is to say, when we use the word "electricity", we aren't actively calculating electric + ity = electricity each time we use the word. Another example of a suffix that used to be productive that is no longer productive is -th (as in "warmth", "length", "width"). We don't make new words with this suffix. So, many of the suffixes we see in older words are no longer productive, but the word is still part of our lexicon.

Phonological restrictions

In other cases, two suffixes can be productive, but be restricted by phonology (that is, syllable structure and word stress). For example, the suffixes -ize and -(i)fy both are used to create verbs, but -ify is restricted to one-syllable words, two-syllable words with an unstressed final syllable ending in a vowel (with a few rare exceptions, e.g. "solidify"). For example, "glorify", "testify", "pacify". Of course, -ize is used in the other cases, so we get "revolutionize", "customize", etc. People make this choice when creating new words without even thinking about it – one will just sound "right". This is one of the awesome things about language! Also, referring back to the above paragraph, these phonological restrictions may change over time and one suffix might take over in all new cases. (This comes from Marchard 1969.)

Semantic restrictions

Many times, a suffix is chosen because it carries certain semantic features, like the suffix -er, which converts verbs to nouns, but also adds the meaning "one who does X". So another verb-to-noun suffix might be productive but doesn't carry that meaning.

Pragmatic restrictions

Sometimes, one suffix works better in a certain situation; think of formal/informal distinctions. I might say "naivety" if I wanted to sound very formal or educated, or I might say "naiveness" in a less formal situation. Another example is that we have the word "neurological", but it is my understanding that neuroscientists often use the word "neurologic" – the alternate word distinguishes the sense of the layman's term from the nuances of the field-specific term.

Sometimes it is unpredictable

Sometimes, a word+suffix combo gets fossilized with a certain meaning and the same word with a different suffix gets fossilized with a different meaning. Think "historical" (happened in the past) vs. "historic" (extra meaning of having a great impact on history -- can be said about present-day events). It's not -ic or -ical that is adding this extra meaning, it's that the words have fossilized in the language and then developed different nuances.

Addressing some of your examples

The word excessive was first cited at the end of the 14th century, so this is a fairly old word. The -ive suffix came from French originally. This suffix distinguishes itself from others on semantic grounds, as explained in the OED (emphasis mine):

the suffix is largely used in the modern Romanic languages, and in English, to adapt Latin words in -īvus, or form words on Latin analogies, with the sense ‘having a tendency to, having the nature, character, or quality of, given to (some action)’. The meaning differs from that of participial adjs. in -ing, -ant, -ent, in implying a permanent or habitual quality or tendency: cf. acting, active, attracting, attractive, coherent, cohesive, consequent, consecutive.

As for the non-words, excessic/excessal/excessical, the most likely reason they were not created is because excessive was already fossilized in the language and, at least thus far, there has been no special nuance or meaning that has demanded an excessic/excessal form that would set it apart from excessive, or cause it to replace excessive. This disallowance of synonymous coinings is also sometimes called Blocking (Aronoff 1976) or the Avoid Synonymy Principle (Kiparsky 1983). It is something that is still being worked out by morphologists today.