[SalesForce] Getting heap size error while deserializing json response in apex

I am using third party service to get data using rest API. But sometimes it return more than 6 MB data. They don't have API to provide data in chunks. I am already using future method to increase heap size limit to 12 MB.

With below code, I am facing heap size error at the time of de-serialization of data. I have also emphasised that line where I am getting error.

@future (callout=true)
public static void CreateProjects(set<ID> setOfProjectID)
{
    HttpRequest request;
    HttpResponse response = new HttpResponse();
    Map<Id,Opportunity> mapOfIdAndProjectToCreate = new Map<Id,Opportunity>([SELECT Id, Name, projectId__c FROM Opportunity 
                                                    WHERE ID IN :setOfProjectID]);
    Integer customerId;
    List<Opportunity> lstOppToUpdate = new List<Opportunity>();
    if(!mapOfIdAndProjectToCreate.isEmpty())
    {
        for(Id OpportunityID : mapOfIdAndProjectToCreate.keySet())
        {
            Opportunity oppCreated = new Opportunity();
            oppCreated = mapOfIdAndProjectToCreate.get(OpportunityID);
            try
            {
                String projectId;
                String requestBody = createRequestBodyForProject(mapOfIdAndProjectToCreate.get(OpportunityID));
                HttpRequest req = CreateHttpRequest(ConstantCls.CreateProjectURL,requestBody,'POST');
                Http http = new Http();
                HTTPResponse resp = new HTTPResponse();
                if(!Test.IsRunningTest())
                {
                    resp = http.send(req);
                    System.debug('heap size after api call '+Limits.getHeapSize());
                }
                if(resp.getStatusCode() == 200)
                {
                    System.debug('heap size before deserialization'+Limits.getHeapSize());
// Till now we have 6 MB data in HTTPResponse resp variable, but when we go for deserializing by fetching its body, it gives Heap size limit error.
// In execution of below line, I am getting Heap size limit error.
                    **Map<String, Object> mapKeyVal = (Map<String, Object>)System.JSON.deserializeUntyped(resp.getBody());**

System.debug('heap size after deserialization '+Limits.getHeapSize());
                    Map<String, Object> data = (Map<String, Object>)mapKeyVal.get('data');
                    projectId = String.valueOf(data.get('projectId'));
                    oppCreated.projectId__c = projectId;
                    lstOppToUpdate.add(oppCreated);
                }
            }
            Catch(Exception ex)
            {
                System.debug('Error In Sync - CreateProjects method Updated - '+ ex.getMessage() + ex.getStackTraceString());              
            }
        }
    }
    UPDATE lstOppToUpdate;
}
public static HttpRequest CreateHttpRequest(String apiUrl, String requestBody, String method)
{
    Map<String,String> apiConfigs = GetAllConfigSettings();   
    HttpRequest req = new HttpRequest();
    req.setEndpoint(apiUrl);
    req.setMethod(method);  
    req.setTimeout(120000);
    req.setHeader('Content-Type', 'application/json');  
    req.setHeader('Authorization', apiConfigs.get('Access Token'));  
    if(String.isNotEmpty(requestBody))
    {
        req.setBody(requestBody);    
    }
    return req;
}

Best Answer

A solution (such as the one your are using) that converts the entire JSON string into objects requires enough heap space for both the JSON string and the resulting object graph. So this is one of the few occasions where using the JSONParser class instead could help, especially if you are only interested in a few of the pieces of data. Using that class you can examine the tokens in the JSON and only keep the ones you are interested in and so (potentially) greatly cut down on the space used for the resulting object graph.

An alternative way to perhaps achieve the same result would be to use JSON.deserialize(jsonString, apexType) and leave out most of the fields on the Apex class whose type you supply but whether that works depends on how the method is implemented internally which is not documented.

Obviously if the JSON string consumes the heap you are still stuck.

(A last resort would be to build a proxy service on a platform that doesn't have such tight constraints such as Heroku and call that proxy service using some "chunk" identifier of your own design. That proxy service can then make the request per chunk to the current end pint and only return the relevant part.)

Related Topic