Present perfect continuous, aka present perfect progressive - expresses an action that recently stopped or is still going on. It puts emphasis on the duration or course of the action.
Although it doesn't necessarily imply the action has finished, it very specifically contextualises the "duration or course" of that action relative to the present moment. Often, to indicate that the action has been continuous right up until [about] now, and has either just ended or will end very shortly.
In fact, unless the context or specific verb suggests otherwise, it more commonly implies that the action is still ongoing. Thus for example "I have been living there for years" normally implies you're still living there at the time of speaking.
But we often use this construction when the action has just finished - for example, when answering the telephone you might say "I've [I have] been waiting for you to call". In practice this means exactly the same as "I'd [I had] been waiting for you to call", since the context clearly implies you're no longer waiting.
Sometimes we use it simply to stress continuous action from past through present and on into the future - "I have been doing this all my life, and I'm not going to stop now".
Present perfect continuous (I have been doing sth) can be used whether or not you're still doing it, whereas past perfect continuous (I had been doing sth) always implies you're no longer doing it.
The short answer is that you may employ the simple past perfect to express a continuing action only when the expression is atelic or bears in context a reasonably natural atelic interpretation.
A telic expression is one which has a goal or ending point "built in" to its sense—finish, for instance. Employing the test suggested in the article linked above, it makes perfect sense to say He finished in an hour, but not (normally) He finished for an hour.
Expressions which do not have such a goal are atelic. In your first example, work is an atelic expression: using the same test, He worked for an hour is acceptable, but not (normally) He worked in an hour. Atelic expressions are, so to speak, inherently continuous. Consequently, a simple past perfect construction use supports a continuous sense; this is why the two are "interchangeable".
Your other examples, however, are telic. Eating dinner and cleaning a room are not (normally) protracted indefinitely, they come to an end when the dinner is consumed and the room is clean. Consequently, using simple perfect constructions implies completion, and if you want to convey that the action continues you must employ a progressive construction.
Note, however, that "telicity" is a very subtle matter in practice. As the linked article tells you, grammarians are in some disagreement over just how it works; and I have been careful to qualify all my analyses with the (normally) tag.
Note, too, that there is an alternative to the two constructions you illustrate. The past progressive ("I was eating dinner when ... " and "I was cleaning my space when ... ") is more natural to my ear than the past perfect progressive. You want the past perfect progressive only if you employ a qualifier like since dawn, which removes the focus from the present-in-the-past to the past-in-the-past, the stretch of time which preceded the present-in-the-past.
Best Answer
This is a somewhat facetious sentence. The meaning is that for four months, John has been going around every day saying, “I’m going to have finished my novel by tomorrow”, i.e., by the next day, he’ll have his novel all finished.
When ‘tomorrow’ comes and he still hasn’t finished the novel, of course, his statement can be related in indirect speech as, “He said that he was going to have finished his novel by today”.
After a while, someone less than kindly points out that John has now been “going to have finished” his novel “by today” for no less than four months—rather a distant use of the word ‘tomorrow’.
You cannot quite simplify the sentence as “For four months now John has been going to finish his novel by tomorrow”, because of that little preposition by. By indicates here a point in time when a certain action is already in a state of completion (has already been completed). It does not indicate the the time when the action is completed—we would use no preposition at all for that, at least not when the time is an adverb-like word1 like tomorrow or today (or something like at or on for regular nouns).
Thus, if he had said “I’m going to finish my novel tomorrow!”, then he would have been talking about the point in time when the novel is finished, and you could similarly simplify your sentence to “For four months now John has been going to finish his novel today”. But if he originally said that he was going to “have finished my novel by tomorrow” (i.e., he might finish it tonight, or it might not be until tomorrow morning—but when the period tomorrow ends, the book will exist in a state of completed finishedness), and you report this faithfully, then you have little choice but to keep all the (deliberately) heavy and clumsy auxiliaries.
1 Various theories place words like today and tomorrow in different categories: adverbs, noun, determiners, even pronouns. The details are unimportant here, just know that they function a bit different from regular nouns.