Syntactically, OP's first alternative (Could not you help him?) is more "correct", in that it places the negating not closer to could - the element it most applies to. But idiomatically today we almost always contract that to...
Couldn't you help him?
...even in "formal" contexts. But people often think of such contractions as "informal", and precisely because everyone normally makes the above contraction, the full form as given by OP tends to sound "strange" to the native ear. Thus, when looking for more formal phrasing we're likely to think in terms of discarding the contraction, but we avoid OP's #1 and go for #2 because it doesn't sound quite so odd.
In fact, given a formal setting (lawyer questioning witness, say) there's nothing at all unusual about...
"Is not this the murder weapon, Professor Plum?"
"Could not the gun cabinet have been left unlocked, Colonel Mustard?"
"Was not the dagger kept with the other cutlery in the kitchen, Mrs Peacock?"
OP's idea that the position of not determines whether it's a genuine enquiry or a complaint isn't quite right. What matters is that a complaint would stress the word not - which you can't do if it's contracted. You want that stress because effectively all the other words in the utterance apart from could and not simply describe the current situation (although less common, in most contexts it's possible to stress could instead of not when the intention is to complain about something rather than ask if it's true).
Note that it's not always possible to distinguish a question from a request. For example, given just...
"Could you not do that?"
...it can often be effectively impossible without further context to establish whether the speaker is asking you to stop doing something (complaint) or asking whether you're unable to do it (question).
In short, context and intonation are crucial in such negated constructions. Consider, for example,...
"Could you not do that?"
...which might mean...
1: You're doing it right now, and I want you to stop
2: It's what you plan to do, and I'm asking if you will (or are able to) change your plan
3: It's not what you plan to do, but I'm asking if you will change your plan
(and probably other possibilities)
If it's not obvious how #2 and #3 apply, consider "Could you not work tomorrow?", which could either be asking you to take an unscheduled day off, to work on a day when you normally wouldn't.
"Dogs, not cats"
is not a sentence: it is a contraction of a sentence. A fuller sentence would be (for example):
"I mean dogs, not cats."
That, in turn is a contraction of:
"I mean dogs; I do not mean cats."
Hence the "not" comes from association with the omitted verb.
The technical term for this is ellipsis; see NVZ's answer.
P.S. In the above answer, I have used the word 'contraction' in its normal, every-day usage to mean making "something … smaller or shorter" (see Cambridge Dictionaries Online). It does not refer to the specific linguistic meaning of 'contraction' cited by NVZ.
Best Answer
For academic students of English, and language teachers - from a post I wrote for ELU
Native English speakers strongly prefer negating the verbs think, believe and want, amongst others, to negating the complement clauses that they license. So, for instance, all other things being equal, we prefer:
to:
We also would tend to prefer:
to:
and there is absolutely no doubt that:
is far more customary than the rather stilted:
Notice that what is implied by (1) is the same as what is literally encoded in (2). However, (1) does not in fact strictly semantically encode the same information as (2) at all. If we made no further pragmatic assumptions about what the speaker of (1) intended to convey, then the maximum we should be entitled to decode is that the speaker does not possess a positive creedal attitude about the existence of Yetis. It is entirely possible that the speaker may have no definite opinion about the existence or non-existence of Yetis, in which case they would not be able to truthfully commit to either a belief or disbelief in them. This might be due to an agnostic state of mind, or it may be merely because the speaker has never even thought about it. To commit the speaker of (1) to a belief in (2) is potentially doing them a great disservice.
Be that as it may, most listeners would understand (1) as conveying the same as (2), and they are indeed entitled to, because most speakers - unless they were wishing to be very explicitly technical about it - would prefer the former to the latter to convey the very same information. What is interesting here is that speakers are modifying the verb denoting the action of belief in order to manipulate the listener’s understanding of the object of the belief - the information in the complement clause. More specifically they are negating the verb denoting the believing, but implying a negation of the complement clause.
This phenomenon is known as SUBORDINATE NEGATION IMPLICATION. Verbs that tend to generate such implicatures seem to be verbs that denote states of intention, epistemic stance or opinion, or those which can be used performatively for advice. Dynamic verbs which denote actions, changes of mental states and so forth do not tend to generate these implicatures. Compare the following sentences with the dynamic verb say:
Here the two sentences do not convey the same information at all. We are not likely to infer the information in the second sentence when we read the first.
One more factor comes into play here. Verbs that generate subordinate negation implicatures, tend to be what are described in the CaGEL as medium strength verbs. They contrast for example 'stronger' know with 'medium strength' believe. The reason that these verbs tend to generate such implicatures is merely that, pragmatically, it does not seem very informative to tell somebody that you don't have a medium strength stance about something. We tacitly infer, on this basis, the more informative proposition that the speaker has a stance about a negative idea.
However, with so-called stronger verbs, on the other hand, it is informative to convey that your confidence in a stance is not 100%, or contrastingly with weak verbs to convey that that not even the slightest positive attitude is given to the proposition in the complement clause. The strong and weak usages of the following verbs do not, therefore, generate subordinate negation implication:
In relation to the original poster's question, it is fair to say that when we are not speaking in a very technical fashion indeed, that if we understand something as having a negative subordination implication, it probably has one. In other words the speaker was intending the listener to understand precisely that the content of the subordinate clause should be read as being negated. However, the original poster's debating partner was technically correct that when we negate verbs such as think, believe, want, what we say does not semantically entail the same thing as the negation of the proposition in the complement clause.
As to why speakers actually prefer to negate verbs such as want and believe rather than to negate their complement clauses, I do not believe that anybody knows (- by which I want you to infer that I believe that nobody knows).