There is a Known Issue where if you use System.enqueueJob in code anywhere, and there are at least two test methods in the same class, it can fail with that error, a "Attempted to rollback to the wrong limit tracker" error, or an Internal Server Error (depending on the cause). A DML that fails in this class will fail with "Attempted to rollback to the wrong limit tracker" when "allOrNone" is "true", or Internal Server Error when "allOrNone" is false. The only two workarounds are to (a) not run System.enqueueJob inside a test method (guard with Test.isRunningTest()), or (b) use at most one test method per class that uses a Queueable interface class. Calling Test.startTest() or Test.stopTest() in a test method that is in an affected class will give the System.FinalException error.
Looking at the error message
You can't edit tab settings for AddOnLicense, as it's not a valid tab.
I found that the issue is the content of profile.tabVisibilities. I could not figure out exactly why the error is happening, but I found a workaround simply omitting the entire profile.tabVisibilities by setting it to an empty list like this:
MetadataService.MetadataPort service = new MetadataService.MetadataPort();
service.SessionHeader = new MetadataService.SessionHeader_element();
service.SessionHeader.sessionId = UserInfo.getSessionId();
MetadataService.Profile profile = (MetadataService.Profile) service.readMetadata('Profile', new String[] { 'Admin' }).getRecords()[0];
MetadataService.ProfileTabVisibility[] tvs = new MetadataService.ProfileTabVisibility[]{};
profile.tabVisibilities = tvs;
MetadataService.ProfileFieldLevelSecurity fieldSec = new MetadataService.ProfileFieldLevelSecurity();
fieldSec.field='Account.Test__c';
fieldSec.editable=true;
profile.FieldPermissions = new MetadataService.ProfileFieldLevelSecurity[] {fieldSec} ;
service.upsertMetadata( new MetadataService.Metadata[] { profile });
Now the result of the last line looks way better:
[ {
"success_type_info" : [ "success", "http://soap.sforce.com/2006/04/metadata", null, "1", "1", "false" ],
"success" : true,
"fullName_type_info" : [ "fullName", "http://soap.sforce.com/2006/04/metadata", null, "1", "1", "false" ],
"fullName" : "adm 2",
"field_order_type_info" : [ "created", "errors", "fullName", "success" ],
"errors_type_info" : [ "errors", "http://soap.sforce.com/2006/04/metadata", null, "0", "-1", "false" ],
"errors" : null,
"created_type_info" : [ "created", "http://soap.sforce.com/2006/04/metadata", null, "1", "1", "false" ],
"created" : false,
"apex_schema_type_info" : [ "http://soap.sforce.com/2006/04/metadata", "true", "false" ]
} ]
I verified that the tab visibility is not messed-up for the Admin profile at least. IMO it would be nice if it was possible to just readMetadata() some stuff, make changes and upsertMetadata() it again - but it seems that some elements (here profile.tabVisibilities) are not update-able and need to be removed.
Best Answer
The Apex testing framework is built to test functionality implemented in Apex.
But there is a lot more functionality in an org than just Apex, so a natural extension of this test functionality is to want to begin to test other functionality. In some cases, the natural features of the Apex test framework align quite closely.
Take for example workflow actions like field updates and assigning tasks. Because these are both data-oriented features, you could easily test the functionality of those features.
But many features do not align with Apex test functionality, most especially configuration/metadata changes.
Furthermore, the testing framework works primarily in the context of the runtime transaction model and is designed to leave no footprint either inside of the org (ringfencing of real data, no commits to data changes, and automatic rollback of DML) and in systems that are integrated with the org (no callouts allowed).
I think the tests you propose are a very good idea. I just think you will need to use an external testing framework to do so. With greater and greater support of config changes in the metadata and tooling API, these kinds of processes are becoming easier to automate.
Perhaps you could adapt your existing build process to perform this test. In other words after you deploy to your sandbox/trialforce org, automate a follow on process to deploy some metadata changes and then rerun all tests.