Don't overthink this

Breathe easy - you cannot do this wrong. Some DMs will favor ability checks over contests and some will lean more heavily on passive over active checks: each of these DMs is playing the game by the rules because "the DM decides what happens".

How to Play

First of all, focus on the fundamental rule of D&D 5e:

1. The DM describes the environment.

2. The players describe what they want to do. ...the DM decides what happens, often relying on the roll of a die to determine the results of an action.

3. The DM narrates the results of the adventurers’ actions.

(PHB p.6 "How to Play", paraphrased)

Note that rolling an ability check is never a requirement. In general, ability checks are only called for where:

• the result is uncertain. Convincing the nothic to do what it wants to do doesn't require a roll. And
• failure is an option. That is to say, failure presents new and interesting options to the players that are significantly different to what success would provide. Failing to convince the nothic not to eat you is going to lead to a combat which is different from not having a combat. And
• there are no re-runs. If the characters' can just keep trying until they succeed then just let them succeed. Failing to convince the nothic not to eat you means that its going to try and eat you right now.

Ability Checks versus Contests

Page 174 of the Player's Handbook calls out the distinction between an ability check which is made against a DC set by the DM and contest where each creature makes an opposed ability check.

Ability Check

For every ability check, the DM decides which o f the six abilities is relevant to the task at hand and the difficulty of the task, represented by a Difficulty Class.

Contest

Sometimes one character’s or monster’s efforts are directly opposed to another’s.

Both participants in a contest make ability checks appropriate to their efforts. They apply all appropriate bonuses and penalties, but instead of comparing the total to a DC, they compare the totals of their two checks. The participant with the higher check total wins the contest. That character or monster either succeeds at the action or prevents the other one from succeeding.

It is worth noting that the outcomes of contests are more random since they involve at least 2 rolls (more if advantage/disadvantage).

Passive Checks

In addition to that, p. 175 introduces the idea of a passive check:

A passive check is a special kind of ability check that doesn’t involve any die rolls. Such a check can represent the average result for a task done repeatedly, such as searching for secret doors over and over again, or can be used when the DM wants to secretly determine whether the characters succeed at something without rolling dice, such as noticing a hidden monster.

The goblins

The author is suggesting:

1. A contest: the characters' Wisdom (Perception) is being opposed by the goblins' Dexterity (Stealth).

2. A passive check: presumably because this is specifically the example given as to when these are appropriate, "noticing a hidden monster".

Equivalently, the DM could call for the players to make an active Wisdom (Perception) check and use the Goblins' passive Dexterity (Stealth): this is mechanically equivalent. Note that an active vs active check is not mechanically equivalent - an active check improves the characters' chances by 2.5%.

The advantages of the author's suggestion are:

• the player's remain in ignorance of why the DM is rolling dice - this can ratchet up tension if you get in the habit of occasionally rolling the dice and making "hmmmm" noises even when there is nothing to roll for. In contrast, asking the players' to make a Wisdom (Perception) check clue them into the fact that there may be something they are missing - unless you also occasionally call for these for no reason as well.
• Active checks are appropriate when the characters are being, well ... active. Be alert for when this happens "I look into the bushes" should trigger an active check. Active checks should never give worse results than a passive check: actively looking for you missing sock is not going to decrease the chance that your eye will light on it by chance.

The nothic

Is a roll called for? If the characters offer the nothic what it wants at trivial cost why would you roll? Other than the fact that the damn thing is insane and you may want to model this with randomness.

What you should determine is if this is a contest or not. It becomes a contest when "one character’s or monster’s efforts are directly opposed to another’s". In a negotiation this is not, or not necessarily, the case. If you decide that it is a contest then a Charisma (Persuasion) by the character is opposed by a Wisdom (Insight) by the nothic with advantages/disadvantages applied depending on the inclination of the nothic to do what they want anyway. If you decide that it isn't then just set a difficulty representing how hard it is to persuade and insane creature to do whatever is being asked.

Passive checks are inappropriate here.

Generalizing

• Don't roll unless you have to.

• A contest is appropriate when there is active opposition and the skill of the opponent is relevant to the outcome. For example a Charisma (Deception) check is more likely to be a contest then a Charisma (Persuasion) check.

• Passive checks are appropriate where a creature is not actively engaged.

• If there is a passive check and an active check it is irrelevant which is which. An active Dexterity (Stealth) check vs a passive Wisdom (Perception) is exactly the same as a passive Dexterity (Stealth) check vs an active Wisdom (Perception). This allows you to let the most appropriate person roll the dice: the DM if you are keeping secrets or the players if you want them more engaged in the outcome.

Basic

I'm going to address the general case rather than the highly specific example in the question. General guidance will be more useful to other readers with a similar question.

That said, let's start with the rules text...

If neither side tries to be stealthy, they automatically notice each other. Otherwise, the DM compares the Dexterity (Stealth) checks of anyone hiding with the passive Wisdom (Perception) score of each creature on the opposing side. Any character or monster that doesn’t notice a threat is surprised at the start of the counter.

By definition, the whole group of player characters are trying to surprise the monsters, or none of them are. The less-capable ones can remain stationary and further away to facilitate that surprise, but they're still involved and still need to roll. The surprised/not surprised status is determined before initiative is rolled.

The reality is that the entire party is trying to be stealthy - some are just trying to do it while moving. The thunky rear guard group still needs to make Dexterity (Stealth) checks, but there are some modifiers that should be applied - advantage and disadvantage.

• The complete lack of motion could grant the folks hanging behind advantage on their checks - in all likelihood simply washing with the disadvantage from the armor they're probably wearing.
• The distance and obstructions could impose disadvantage on the passive Perception of the enemies, but only against the rear guard and only if they're far enough back that the distance is a hindrance to the targets. If they're just outside the room, no disadvantage should apply. If they're down the hall or around a corner, then it just might.

In effect, the advance team rolls Stealth with a DC of the targets' Passive Perception. The rear guard rolls Stealth with advantage against a DC equal to the targets' Passive Perception - 5.

From there, simply follow the normal rules for surprise. What they can do when the round begins can vary - maybe they don't have enough movement to get close enough to do anything, maybe they're forced to dash, maybe they have to use ranged attacks. There is no "Surprise Round" in 5E, as in previous editions.

Advanced

Because the rear guard is also ignorant of when the advance team actually strikes, it may be desirable (or at least somewhat more realistic) to consider them as three "sides" in the engagement, despite the fact that two are naturally allied.

Compare the rear guard's Passive Perception (you may consider giving them +5 for advantage because they know something is coming, just not exactly when) to the advance team's Stealth rolls, too, then follow the normal rules for surprise. This makes it possible for the rear guard to be surprised during the first round, making them a bit slow to engage because they have to wait for the sounds of conflict to know when to move in.

Alternatively, another method to reflect the rear guard's imperfect information would be to impose disadvantage on their Initiative checks. This could be interpreted or explained as them waiting until they're absolutely certain they heard the fight begin before moving in.

Advantage & Disadvantage

Advantage and disadvantage are among the most powerful tools at the DMs disposal for rewarding player thoughtfulness. They exist as a quick way to grant a benefit or drawback if the situation is good for a given participant. It has an almost minimal impact on speed of play, but can make players feel like their non-mechanical ideas have impact. What the exact advantage is doesn't matter. What the exact disadvantage is doesn't matter. If they have a big enough one to matter, apply it.