Using "curious" in the sense of "odd" or "strange" is more British than American. It does still crop up here and there, but is somewhat archaic, formal -- or, in fiction, intended to convey a British and/or old-fashioned kind of person. E.g., 'How curious!' Dorothy said, peering at the bright purple tree.
So absent context, "The curious boy" will be ambiguous. No getting around it. You have to tell whether it's about the boy investigating something ("The curious boy pressed deeper into the forest") or about a boy being odd in some way ("The curious boy walked down the sidewalk, and we all stared at him"). Or if it means both!
This ambiguity will also be used when someone wants to make puns. "What a curious creature you are!" may mean "you are always wanting to investigate" or it may mean "odd." I recall there being some of this double-meaning in Alice in Wonderland.
"He's very curious" will generally be understood to mean someone has the quality of curiosity, not oddness, in modern American English, but if you want to reduce ambiguity, you use "about" to clarify. "He's very curious about everything!" "She's very curious about ancient Egypt." "The neighbors very curious about our business, and I want better curtains in our house!"
In modern American, "curious" people or animals (the curious cat, the curious child) will be "inquisitive" with an ambiguous side-order of "odd"; the more stereotypically inquisitive the animal (or person: the curious neighbor), the more likely the assumption will swing to "inquisitive." But "curious" inanimate objects will always be "odd." And "a curiosity" is an odd thing.
(You wouldn't use "Curious you!" because either way, applying adjectives to pronouns rarely works well. "Red you!" "Angry him!" "Stinky her!" "Inquisitive me!" -- you can get away with it sometimes, in a slangy fashion, but you tend to need a noun, like "Curious George," the monkey who is curious about everything! (The Curious George books are a series of children's books which strongly normalizes "curious" as an adjective meaning "inquisitive.")
So if you want to avoid ambiguity, use "odd" or "peculiar" or "weird" or "fascinating" in American English, to mean those things. And only use "curious" for things like "He's very curious" (inquisitive) or "I'm curious about this sentence here" (I want to understand this sentence better, I am interested in this sentence).
According to http://www.worldwidewords.org/qa/qa-unv1.htm words with a Germanic root use the un- prefix and words with a Romantic root use the in- suffix.
There are also the prefixes a- as in asexual and non- as in nondescript.
Unless you're a professor of linguistics and etymology I think your best bet is to use un-, which is the most common prefix, and learn which words use a different prefix on a case by case basis.
EDIT: I know this doesn't completely answer your question for why unstable turns into instability, but the truth is that although there is a general rule for which prefix to use, there are so many exceptions that it's pretty much arbitrary (without reason).
Best Answer
According to the Cambridge dictionary, uttermost is the formal version of utmost, which suggests that utmost was originally regarded as some kind of abbreviation.
You will find a definition of the origins of utter here.
Languages were not designed by a team of engineers: they have evolved over millennia. As with humans, evolution values diversity: it does not discard redundant or duplicated features, and only discards with extreme reluctance those that have a negative impact.
English developed over a large area in terms of the communications available at the time, and so duplication and inconsistencies were bound to develop. Once they exist, the evolutionary force that drives change in language is in no hurry to get rid of them.