I’m gonna have to disagree with @prash, and so I’ll post this as a competing answer, offering an alternative take on The Rule:
The prefix mono- comes from Greek monos, itself rooted in the Proto-Indo European *men- (small). uni- comes from the Latin unus, itself from PIE oinos (and thus related to the Greek oinos, “ace on a dice”; Ancient Greek otherwise used alpha for the number one).
Forming words from mixed Latin and Greek roots used to be frowned upon. In the classical tradition, you would form things either from Greek or Latin, not mix them together.
As time evolved, a number of Greek and Latin roots entered the English lexicon, mostly as suffixes and prefixes. Because they became part of the language in that way, they could be combined with other words, being somewhat free of the older rules: you wouldn’t go forging a brand new words directly from Latin and Greek roots, but you would add a prefix (say, auto) that came from Greek but was now assimilated into the English language, and put it in front of an English adjective (say, mobile, which came from Latin movere through Middle French). And kaboom, you have: automobile. And few frown upon it.
So, what's the current status? It's now mostly a matter of taste. There is no strict rule, and you should form the word that sounds better to your ears.
Wikipedia calls words formed with roots of different languages hybrid words, though I've never really heard the term used. I’d prefer we call them heteroradicals (tip of the hat to JSBᾶngs).
Moreover, the Wikipedia article linked above has a long list of such English words, many of which are very common: homosexual, television, hyperactive, electrocution, bigamy, Minneapolis, etc. It even includes the following two words which have the mono- prefix:
TLDR — I usually don't do tldr’s, but this has turned into a general post on the topic while you had a very specific question. So, here's the answer to your very specific point: it's a matter of taste, choose the one that sounds best. If the roots are from the same language, as an added bonus it will please some more people.
The Latin suffix -io, stem -ion-, is usually added to the supine stem ( ~= past participle stem) of a Latin verb, in order to form a noun of action (meaning "x-ing"). It is sometimes also added to other stems, but usually not. The verb vocare ("to call"), present stem voca-, supine stem voca-t-, gets vocatio ("calling"); the stem of that noun is vocation-, which is what our suffix was based on.
Because the regular suffix for the supine stem / past participle is -t- in Latin, we often see -tion; but compare also miss-ion, fus-ion, cohes-ion, inflex-ion, etc.. The same suffix can also be found with non-supine stems, though less frequently so, like rebellion, legion, etc.
Now how can we predict what comes before -ion in English? I don't have a neat system based on tight rules, but perhaps I can give a few general directions.
What we could do is try and think of a cognate verb, as you have been rightly doing. Does this verb end in vowel + t or vowel + te? If yes, it is probably based on a (real or reconstructed) Latin supine stem; then -ion can come right after the t, as in inhibit, complete, migrate.
And what if the verb does not end in vowel + t(e)? Then you need to know whether or not its stem is a supine stem in Latin. There is no way to know this except by consulting a Latin dictionary. But there are some unreliable rules.
Irregular supine stems most often end on s or x, so verbs like flex (from verbal stem flect-, "to bend") and fuse (from verbal stem fund-, "to pour") contain the supine stems flex- and fus-, leading to inflexion and fusion. But vex- and pos- are not supine stems, so vexation and position, from supine stems vexa-t- and pos-i-t- (the i is probably a fused theme vowel).
There are many supine stems that end on -pt- or -ct-, like act- and rupt-, leading to action and eruption.
Verbs ending on -nt are nearly always based on the present-participle stem -nt-, and so cannot get -ion right after nt; but there are a few supine stems on -nt- as well, mainly vent- and tent-, leading to -vention and -tention.
But there are also some nominal ("noun") stems that can get -ion, as mentioned before, like mens, "mind", stem ment-, leading to mention; and dens "tooth", stem dent-.
Cohabitation is an apparent exception. But both habē-re ("to have") and habita-re ("to inhabit") exist as verbs. Habitare was actually formed based on the supine stem habit- from habēre; then a new supine stem was formed based on this new verb, habitat-. And so we have pro-hibi-t-ion and co-habi-t-at-ion.
Normally it isn't very functional in Latin to make a new verb based on the supine stem of another, since you already have the original verb; but it is sometimes done, often to add a sense of frequency or intensity, and so two supine forms may come to exist. Usually a is then added after the supine stem to turn it into a new verbal stem, because a indicates a causative verb, i.e. a verb that means not "to do x" but "to cause someone to do x" (with nominal stems, "to turn into x, to affect with x", or simply to turn any nominal stem into a verb). So fugere = "to flee"; fugare = "to make flee, to drive away"; donum = "gift", donare = "to present as a gift" (or simply "to give").
Cf. haerē-re ("to stick") => hae-s-us (past participle, "stuck") => hae-s-ere => hae-s-it-us => hae-s-it-are => hae-s-it-at-us => hae-s-it-at-io =>> hesitation. Notice that the same supine suffix has been added thrice: s, it, at (the t was turned into an s after certain verbal stems; the i and the a are theme vowel and causative vowel, respectively). And so we have co-he-s-ion and he-s-it-at-ion, both from the same verbal stem haer(e)-.
So it is theoretically possible to add -ation after any supine stem; but that is usually not done, because just -ion is shorter. After any stem that isn't a supine stem, just adding -ion is unusual, so we will usually have to turn it into a (real or hypothetical) verb first with -a-, then add the -t-, then -ion — i.e. we have to add -ation. But, as mentioned above, there are many exceptions, where -ion can be added directly to non-supine stems; however, this is a fixed set of words that mostly already existed in that approximate form in Latin.
Best Answer
Yes and no.
Yes because it's possible, but no because technically when you want to do this, you are lexicalizing one of them and then adding on the [pre|suf]ix to that base word and so are no longer strictly using a prefix and a suffix together.
So in the case of mononess, you aren't referring to a singularity of quality, rather the quality of being one/singular, so mono- is lexicalized as mono1 to which the suffix -ness is added. I could perhaps understand someone talking about the preness of something (the quality of coming before, or pre- + -ness), but I think I'd definitely need a hyphen to actually mentally decompose the word properly.
In the case of mononess though, it's a bit hypothetical, since the word oneness is already regularly used for the concept. In fact, I reckon most any combination will likely be better served by some other word or phrasing.
1. I'm ignoring the fact that apparently mono is already an adjective mainly because its glosses are monochrome and monophonic which don't apply here (that and I've neither seen nor used it)