There are essentially two distinct ways of how "to hack" (as a verb) or "hacker" (person who hacks) is used in the context of computing.
The more traditional—and positive—meaning emphasises the cleverness of the activity. Some examples about this use:
- The Hacker subculture article on Wikipedia
- The Jargon File – a glossary of hacker slang, which "illuminates many aspects of hackish tradition, folklore, and humor"
- People who work on the Linux kernel (generally speaking very smart and capable programmers), are commonly called "(Linux) kernel hackers".
In this context, another word, cracker (see definition 8), is often used in place of "hacker" to refer to those who "break computer security without authorization".
The other—negative—meaning, which is probably more recent (but not at all new) and more widespread, emphasises the illegality or lack of permission. In other words, "hacker" in this sense is the same as what "cracker" means in the hacker subculture mentioned above. :-) Example:
- The Hacker (computer security) article on Wikipedia states that "In common usage, a hacker is a person who breaks into computers and computer networks"
And yes, as you suggest, "to hack" can indeed also mean "to put together something that is clumsy and inelegant yet (somewhat) functional". (And as a noun "a hack" means the result of such work, i.e. a quick, somewhat ugly (possibly temporary) solution to a programming problem.)
But note that "hacker", to my understading, is rarely used in this sense of "to hack", i.e. to mean "someone who does quick inelegant solutions".
To sum it up, all three meanings you described indeed exist, and if you use the word you should be careful to establish the context clearly to avoid misunderstandings.
When talking with people outside the IT / software industry, "hacking" will most likely be understood to have the negative connotation (cracking illegally into systems). On the other hand, be aware that if faced with an audience full of (bearded) UNIX users, using the word "hacker" in the negative way could well earn you their disrespect. :-)
"See if you can't" and similar expressions are a slightly old-fashioned but still common idiom in British English.
I cannot find any definite difference in meaning from "see if you can", except perhaps a hint of the vagueness that is used to add a little politeness (like "would you be able to?" for "will you?").
Edit: missed your last question: no, this is limited I think to verbs of finding out ("see" in that sense, "check", "find out", "work out"), uncertainty ("wonder" and perhaps "not be sure") and permission ("ask" in the sense of asking permission). I don't think it's used with "doubt", "tell" or "know", or with "ask" when asking about information rather than permission. It also sounds odd with more formal verbs like "verify", "determine" or "ascertain".
"Doubt" is an oddity, because it would seem to fit the category of uncertainty, but "I doubt that/whether he won't go" doesn't sound right to me.
Best Answer
The word subtle is usually construed to mean "hard to understand due to intrinsic difficulty". It would be inappropriate to use that word to refer to a course or a subject which is hard to understand due to the incompetence of the teacher.
Subtle is sometimes used to describe something which is overly clever, and therefore wrong or misleading. But this semi-ironic usage of the word is itself pretty subtle (heh), and depends heavily on context. In any case, this doesn't sound like the word you want for your context.