[RPG] Are random encounters supposed to supply treasure

I think OP is trying to account more for the fact that a larger group of adventurers reasonably will attract more attention than a smaller group. This isn't a tailoring of difficulty per se, but more of an expression of the amount of "noise" the adventurers make going through hostile territory.

In this case I would advocate rolling multiple times on the encounter table. A nonlinear buildup would probably be accurate, so let's say we roll a number of times equal to the square root of the size of the party, which is more or less the party's cross-section if you imagine them as a vaguely circular blob moving through the region.

This means that a party of 2-3 adventurers gets encounters about as often as the rules suggest. A party of 4-8 adventurers gets encounters roughly twice as often and sometimes they run into twice as many bad guys.

You shouldn't make the check in the first place.

Unless they are following a path, or something like it, adventurers traveling in the wilderness run the risk of becoming lost. The party's navigator makes a Wisdom (Survival) check when you decide it's appropriate. (DMG p.111, "Becoming Lost," emphasis mine)

And let's remind ourselves of the UA Ranger's class feature:

Natural Explorer

• [...] your group can't become lost except by magical means [...].

(N.B. in the UA version, rangers have this in all terrain; there's no more concept of "favored" terrain. Or, if you like, all terrain is favored.)

I argue, then, that the group is following "something like a path": their ranger! Or, at least, that the GM should never decide it's appropriate to issue a 'becoming lost' check to a party with a ranger. I mean, they've got so little...

But let's say you do get lost:

Clearly by magical means. This didn't happen by means of a failed check, so there's no need to assume that all three consequences enumerated for failed checks happen. Let's consider them severally:

1. ...inadvertently travels in the wrong direction. Well, you didn't. Unless you became (magically) lost without realizing it and continued traveling in the wrong direction. Then you did. GM's ruling on how the magical-lostness works.
2. ...becomes lost. You are.

3. The party's navigator can repeat the check after the party spends 1d6 hours trying to get back on course. So a party without a ranger has to wait 1d6 hours and pass the check to stop being lost. It should somehow be easier for the party with the ranger.

   But remember: the only reason they're lost in the first place is because, magic. (Not only magic, but magic a GM made up. I can't find any spell or item in the PHB or DMG that does it.) So you're squarely in the territory of "you make up how to get unlost."

   Perhaps you rule that after 1d6 hours there's no need for a check, because the ranger de facto passes all checks. (That's how you've internalized Natural Explorer.) Perhaps you rule that the ranger has to pass a check, but it's after 1d4 hours, or 1 hour, or lowest 1 of 2d6 hours. (Some way to represent a pseudo-advantage on that time-roll.) Or maybe the party stays lost until you say they're not, because magic.

   Just make sure it's a little easier to get unlost because of the ranger. After all, combat and social scenes are harder for the party because of the opportunity cost of carrying a ranger around; exploration should be a little easier.