Apex – Salesforce Bulkification Across Transactions

apexbulkificationgovernorlimitstransactionvisual-workflow

Here is a fun question I have been puzzling over.

So, I have been learning a lot lately about salesforce bulkification and governor limits, specifically as they relate to bulk upload of data into salesforce. One thing has me confused. Salesforce performs bulkification when uploading lots of data. My understanding of this is that it splits the large data set into batches of 200 records each, and applies govern limits to each of these transactions separately (rather than treating the entire upload as a single transaction, as a large data set could easily fail the governor limits).
The other thing is that, in the context of a given transaction, salesforce has bulkification rules that prevent excessive operations. For example, in the context of a flow, the 200 records would be flowing through, then stop at a bulkifiable event (such as a DML element) until all the records arrived at that element, then perform the operation once for all the records in that transaction to conserve for governor limits. Sounds awesome. But this brings me to my question.

I have a process that looks like this:

a custom button on a list view (related list) sends the parent record ID to an invoked flow.
I then pass the parent record ID to a custom apex action.
Within the apex action, I do a soql query to get all the child records, then I loop over the child records.
- For every 10 records, I use the Apex interview class to create a new flow interview for the next flow that will perform extensive logic.
- The very first thing I do in that next flow is call a pause element.
- So for example, if I had 100 child records, the next flow would have 10 interviews created, then immediately paused.
- This in essence gives me 10 brand new transactions (I am doing this to reset governor limits, as the logic I am processing in the next flow is rather extensive and fails if it runs on many more than 10 records).
  - I have confirmed that I now have 10 independent transactions by viewing the Time-Based Workflow queue, and can see that the next flow has 10 pending jobs.

But here is the tricky part. The first thing I do in the paused flow when it is resumed is

call an update records element (which is triggering a record triggered flow that does some logic and record manipulation).
This is a bulkifiable event in salesforce, and I am trying to figure out how the batches are working.
I would EXPECT that since I have 10 independent transactions calling this update records element that the update would be called separately for each transaction (and that the resulting record triggered flow would be called 10 separate times).
However, is salesforce smart enough to recognize that I am only updating 10 records at a time, and somehow combine these update calls together (within its 200 record batch size limit)?
The reason I am confused is that when I open the debug logs, I would EXPECT that if the updates are being triggered independently, I would have 10 separate debug logs (one for each transaction). However, I just have one debug log that is recording governor limit use.
As a side note, the behavior in the log is odd. It is adding up the SOQL governor limits almost to a point of failure (which is why I want the behavior split), but the CPU time is NOT adding up.
Although I THINK that 10 separate transactions should be running, the logs tend to indicate that maybe that is not happening????

In summary, does salesforce bulkify update records flow elements across transactions?

And do I have the correct understanding that a flow pause element would create separate independent transactions when calling flow interviews from an apex loop (for every 10 records)?

I would appreciate if someone was able to clarify some of these details. Also, I have struggled finding any salesforce documentation that answers these questions, so if you could point me to the relevant docs, that would be a plus. Thanks so much!! Here is the initial apex action where I am looping over the records and calling the flow interview for every 10 records:

public class TestInterviewSize {
    public class FlowInputWrapper {
        @InvocableVariable(required=false)
        public String listId;
    }
    
    @InvocableMethod(label='Get Accounts by List ID')
    public static void getAccountsByListId(List<FlowInputWrapper> inputList) {
        System.debug('Invoking method');
        System.debug(inputList);
        FlowInputWrapper input = inputList[0];
        String ListId = input.listId;
        Integer iterater = 0;
        List<List_Entry__c> batchedRecords = new List<List_Entry__c>();
        List<ID> batchedRecordIds = new List<ID>();
        Map<String,Object> inputVariables = new Map<String,Object>();
        
        String flowName = 'Create_Accounts_for_List_Entries_v_2';
        
        
        List<List_Entry__c> listEntries = [SELECT Individual_Email__c, Organization_Website__c, Multiple_Org_Matches_Found__c, Multiple_Individual_Matches_Found__c, Create_Individual__c, Create_Organization__c FROM List_Entry__c WHERE List__c = :input.listId AND (Create_Individual__c = true OR Create_Organization__c = true)];
        
        Integer numberOfListEntries = listEntries.size();
        System.debug(numberOfListEntries);
        
        for(List_Entry__c entry : listEntries) {
            batchedRecords.add(entry);
            batchedRecordIds.add(entry.Id);
            iterater += 1;
            if(math.mod(iterater, 10) == 0 || iterater == numberOfListEntries) {
                System.debug('Testing!!!');
                System.debug(iterater);
                System.debug(batchedRecords.size());
                inputVariables.put('ListId', input.listId);
                inputVariables.put('ListofListEntries', batchedRecords);
                inputVariables.put('ListEntryIds', batchedRecordIds);
                Flow.Interview myFlow = Flow.Interview.createInterview(flowName, inputVariables);
                myFlow.start();
                batchedRecords.clear();
                batchedRecordIds.clear();
                inputVariables.clear();
            }
        }
    }

}

Edit: I am attaching the debug log for reference. Notice that the governor limits are stated quite a few times (especially toward the end of the log). Is it possible that salesforce recognizes that several transactions take place around the same time, and so just generates a single debug log? However, it is odd because the SOQL limitation continues to accumulate throughout the entire log, insinuating / behaving like a single transaction. At the same time, the CPU time limitation does NOT behave this way – it seems to reset often, as it is not accumulating. Not sure if this means I have a single debug log housing multiple transaction (which doesn't make sense with the SOQL limits accumulating), or if salesforce did indeed get smart enough to batch my 10 transactions together (which doesn't make sense with the CPU limits NOT accumulating). https://drive.google.com/file/d/1yb_LXS2EU-m__AadT5Uyq0gYksc8YUPl/view?usp=sharing

Best Answer

One debug log is one transaction. Full stop. What happened here is that you placed a bunch of interviews into a paused state, so they went to the asynchronous queue. When Salesforce had time, it selected up to 1,000 paused jobs that were due for execution, batched them by user/owner, then ran those in groups of 200.

You can see this in the various parts of the log:

FLOW_START_INTERVIEWS_BEGIN|200
FLOW_START_INTERVIEWS_BEGIN|200
FLOW_START_INTERVIEWS_BEGIN|173
FLOW_START_INTERVIEWS_BEGIN|12
FLOW_START_INTERVIEWS_BEGIN|12
FLOW_START_INTERVIEWS_BEGIN|9
FLOW_START_INTERVIEWS_BEGIN|2
FLOW_START_INTERVIEWS_BEGIN|2

As you can see, 610 interviews in total were executed across 8 batches. It appears that some of the limits reset between each batch, but others do not.

Consider using a Queueable or something if you want to ensure your interviews are actually separate transactions.

Simple Demonstration

Here's a demonstration. I put together a simple Flow thus:

It's an autolaunched Flow. It queries a single Account record (at random) and stores it in an sObject variable. It calls an Apex Invocable Method (below) to check Limits.getQueries() and stores the result in a Number variable. Then, it Waits for a Platform Event.

When the event is fired and the Flow resumes, it re-checks the SOQL limit and stores that value too. Finally, it sends an email with both SOQL values and a field value from the original queried Account back to me so I can look at the results.

Here's the Invocable Apex:

public class InvocableSOQLLimit {    
    @InvocableMethod(label='Check SOQL Limit')
    public static List<Integer> checkSOQLLimit() {
        return new List<Integer>{Limits.getQueries()};
    }
}

And here's what I get in my email afterwards:

Account Name: ACME Industries
SOQL Before: 1 
SOQL After: 0

This shows that the transaction limits were reset (we burned 1 SOQL on the original query, and none after the Wait element), but we preserved the original Account field values queries before the Wait. (They may now, of course, be stale).

Of course, you must contrive some way to have your Flow restarted. I just used a sample Platform Event, Test__e, that I fired via Anonymous Apex:

Test__e event = new Test__e();
EventBus.publish(event);

But that's not going to facilitate a production Flow. Instead, you can configure your Wait element to go off "more or less now", zero hours after the current date and time:

As far as I can tell, the docs (and two) are not explicit on the exact semantics of this time trigger when it's in the past, but it seems to work like Process Builder (as one would expect, since they're the same under the hood and the rest of the mechanics are similar): setting it to zero results in a resume within a few minutes, subject to server load.

Salesforce Flow Limitations – Loop vs Batch Records

What is the difference between a transaction and interview?

A transaction is one complete execution unit as far as the server is concerned. I've always explained it as this: one debug log equals one transaction. An interview is one or more transactions that the Flow Runtime executes to process all of the elements.

Do governing limits apply to a transaction or an interview?

They apply per transaction, not per interview.

When I set my scheduled flow to run on a batch of records (therefore causing multiple interviews), how exactly did this differ from my old solution of running a loop that stopped with an error after 10 seconds?

Because each separate execution context is its own transaction. By using Pause/Wait elements or Asynchronous Paths, you're creating multiple transactions to break up the workload.

If bulkification is the cause of the dreaded "Now it works but I don't know why!", why exactly does it eliminate the 10 second limit? Is it possibly creating a new transaction after each step in the bulkification process?

As hopefully obvious by now, because each transaction has its own 10 second limit, so by breaking up the flow, you end up with more CPU time to complete the workload.

Best Answer

Related Solutions

Apex – Breaking a Flow into Two Different Transactions

Simple Demonstration

Salesforce Flow Limitations – Loop vs Batch Records

Related Topic