I'm new to packaging so apologies if this is an obvious question.
When running the unlocked packages tests that reference unpackaged metadata in the SUBSCRIBER ORG, are these tests expected to pass? Or do we anticipate and ignore errors?
I have tests in our unlocked packages that reference unpackaged metadata (Custom Permissions, etc). When I run those tests in our scratch org, they pass. Based on that, I'll create a package and install it in a subscriber org.
When I run the package tests in my SUBSCRIBER org, some of the tests that reference the unpackaged metadata fails. These failures right now are assertion failures.
Since the docs say this, I am leaning towards this is expected behavior:
The unpackaged metadata isn’t included in the package and isn’t installed in subscriber orgs.
Not to get too subjective, but if others have seen this, would like to hear how you design your tests so they don't fail in a subscriber org. The issue is that it creates noise. For these offending tests, one idea is checking if the user is in a subscriber or scratch org, and not running tests in the subscriber.
Best Answer
Those tests are not expected to pass. (Unless, of course, you deliver the unpackaged metadata to the subscriber).
Packaged tests failing in subscriber orgs is really common. You essentially cannot count on any test that performs DML passing, as subscriber-owned customization is highly likely to change their behavior.
It sounds like these tests are likely using Dynamic Apex or Dynamic SOQL, since if they had static references to your unpackaged metadata the package wouldn't install at all. Those types of test are also highly vulnerable in subscriber orgs.
Thing is, your packaged tests don't actually have to pass in subscriber orgs. The subscriber always can use the Run Local Tests option when executing a deployment to ensure that only their non-packaged tests execute. Any other managed packages installed will likely have the same problem as yours, so it's relatively unlikely to create a new challenge for users.