[SalesForce] Code Coverage Calculation – Seems to be including code in test methods

According to the documentation:

Test classes (classes that are annotated with @isTest) are excluded
from the code coverage calculation. This exclusion applies to all test
classes regardless of what they contain—test methods or utility
methods used for testing

However I am currently finding that to be false (not the case).

As you can see in the included image (not all classes included in image) all my non test classes have coverage of 100% with a few stragglers dus to overzealous catch blocks. In total, for non test methods I have a grand total of 27 lines uncovered.

enter image description here

The coverage indicates that 324 lines are uncovered.

In my classes annotated with @isTest there are 297 lines across all @isTest annotated classes where the methods are not marked as test methods (Utility methods).

I have checked the code coverage from the Dev console, IntelliJ, the Setup UI and they all say the same thing 84.xx% total coverage.

I have cleared the test history, recompiled all classes, all the usual stuff.

Is the documentation wrong?

An example method in an @isTest annotated class

private static PageReference setTestPageReference(){
    PageReference pr = Page.Example_Page;
    pr.getHeaders().put('Host','https://login.salesforce.com');

    test.setCurrentPageReference(pr);
    return pr;
}

The class that contains the above method has 0/5 line covered. When I add the testmethod to the method definition, the class changes to have 0 uncovered lines and the code coverage for the entire org goes up by .01%

Basically it seems that code in a @isTest annotated class IS being included in test coverage which is a problem since I have a large test Utility class annotated with @isTest and none of the methods are marked as test methods.

So the Questions:

  1. Am I the only one observing this behavior?
  2. Should I worry about it?
  3. Any harm in adding the testmethod designation to the utility methods? (It feels wrong)

Best Answer

I've had similar conversations about this before with salesforce, because this was actually causing us issues in deployment. What we came away with was as follows.

The Documentation is Correct

According to Support, any class that has @isTest specified should be completely exempt from code coverage. It should display as 0/0 coverage.

The System is Wrong

They confirmed with R&D that someone internally has already logged a bug regarding this. There is a Known Issue that's tracking this bug.

We Should Test (For Now)

In our case, we had nearly 500 lines of code that were showing no coverage, and it was actually causing deployments to fail with a net total of 74% coverage. If you're getting deployment errors, then you need to unit test your unit tests. Otherwise, don't worry about it.