There isn't really any vagueness about when to use must and when to use should.
Must always implies absolute obligation or certainty.
Should always implies a request, suggestion, or expectation although in some contexts a request may be so strong that it could be seen as an obligation.
See @Cerberus's excellent answer here exploring the "built-in slipperiness" of English words commonly used in the general area of volition/expectation - where must applies to both at the extremes of obligation/certainty. Things only really get murky at the lesser levels.
But OP's example 2 is structurally ambiguous - You must cross the street to get to that store could mean any of...
You are obliged/I order you to cross the street [and thereby to get to that store]
You would/will have to cross the street if you wanted/need to get to that store
More naturally we interpret the whole sentence as informational, with an implied if you want to get to the other side. The obligation implied by must doesn't come from the speaker - it comes from the laws of physics which say the only way to be on the other side of the street is to cross it.
Note that in practice, people often deliberately or unwittingly flout these distinctions. I'm sure the vast majority of people who have ever said, for example, "I must be mad!", or "We must have dinner together soon" didn't really mean they were absolutely sure, or imposing an absolute order.
Doubtless as a result of historical changes in meaning and ellipsis, the meanings of prepositions in specific modern-day expressions are legion, often etically unpredictable, and not infrequently apparently illogical. There are broad rules, but they are very broad, and exceptions are many. Avoidance of ambiguity is desirable, but not always achieved: even 'The thief opened the door with the duplicate key.' is ambiguous (instrument (= using), or identification (= which had) of the door?)
David Thatcher, in Saving our Prepositions, writes [re-formatted]:
Greenbaum (103) cites “an empty aspirin bottle was found by the
deceased.” This, he says, “sounds as though the dead person found the
bottle rather than, as was presumably meant, that the bottle was found
beside him.”
The art section of my local newspaper ran the headline,
“the world of ballet has been blessed by many fine composers,”
suggesting that composers, en masse, have been usurping a priestly
prerogative. By, of course, should have been with. The broad
distinction is that by denotes the agent, or essential agent, of an
action, and with the instrument of an action. Compare “he was struck
by the sun” with “the sun struck with its rays, “the tree was shaken
by the wind” with “the wind shook the tree with its strong hands,
“”the city was destroyed by fire” with “he destroyed the city with
fire” (examples cited by Fernald 189).
In practice, by and with
are used less strictly, but “where with or at can reasonably be
used instead of by, they should be” (Greenbaum 103).
That the distinction is not universally made is shown by examples such as
We must do it by long division. [internet]
and
The secret doesn't lie on whether you made it by machine or by hand
but on the embroidery supplies you use to craft it. [internet]
both showing instrumentality.
But in OP's example, 'with' is the accepted choice. This is probably strongly connected with the fact that 'key' is concrete whereas 'by hand', 'by long division' ... show methods (long division being abstract, and hand in this expression intermediate).
Best Answer
I always use by, because in this context we mean "grouped by"
PnL by trader = PnL results grouped by trader
PnL by day= PnL results grouped by day
VaR by asset = VaR results grouped by asset
VaR by portfolio = VaR results grouped by portfolio