[SalesForce] RestRequest RequestBody instead of Params

I am leveraging a webhook from a third party API (Mandrill) that sends me JSON, and have set up a Force.com public site with a RestResource to handle the incoming request. I've been noticing that if the request body gets too big (the JSON is all on a single parameter), it cuts off around ~10k characters (my current request has about ~80k). While this is definitely not the normal size of the payload, when an error occurs it adds subsequent values to the payload for the next retry, and that appears to how I got to this point.

What I'm mostly curious about is whether there is any way to force the use of RequestBody instead of Params. If I could leverage the Blob instead of a String, I shouldn't have the same issue (I tested it by POSTing a request with the same body, but without setting the Content-Type (in this case application/x-www-form-urlencoded) and I was able to see the entire request. From the documentation, it appears that if your request params are JSON, SF will deserialize it into a Map of Strings for you and clear out the RequestBody.

UPDATE: Adding Sample Code

The pertinent request headers:

POST /services/apexrest/Webhook HTTP/1.1
Host:  
Content-Type: application/x-www-form-urlencoded 
Content-Length: 462817
Connection: close 
Accept: */*

Sample Data (extrapolate this to a bigger size, my actual data has sensitive information). The key is there is one parameter, and a lot of data within it (my actual values have about 15 fields per item):

events=[{"event":"test", "msg":"test"}, {"event":"test2", "msg":"test2"}, ...]

Code:

@RestResource(urlMapping='/Webhook/*')
global without sharing class RESTWebhook
{
    @HttpGet
    global static String doGet()
    {
        return 'Please use POST. No data saved.';
    }

    @HttpPost
    global static String doPost()
    {
        RestRequest req = RestContext.request;
        RestResponse res = RestContext.response;

        return PostHelper(req, res);
    }

    global static String PostHelper(RestRequest req, RestResponse res)
    {   
        try
        {
            System.debug('Body: ' + (req.requestBody != null ? req.requestBody.toString() : ''));
            System.debug('Params: ' + req.params);

            // here I can create a new custom records with the params.
            Webhook_Test__c wt = new Webhook_Test__c();
            wt.LongText__c = req.params.get('events');
            insert wt;
        }
        catch (Exception e)
        {
            res.statusCode = 500;
            return e.getStackTraceString();
        }

        res.statusCode = 200;
        return 'Success';
    }   
}

Best Answer

Per RestRequest methods, if the function accepts parameters, it will be deserialized into the provided parameters, otherwise it will be in RestRequest.requestBody. The solution, then, is to make sure your function takes no parameters.

If the Apex method has no parameters, then Apex REST copies the HTTP request body into the RestRequest.requestBody property. If there are parameters, then Apex REST attempts to deserialize the data into those parameters and the data won't be deserialized into the RestRequest.requestBody property.

Update 1

I tested the behavior this morning, using the following test class:

@RestResource(urlMapping='/demoRest')
global class RestDemo1 {
    @HttpPost
    global static String doPost() {
        RestRequest req = RestContext.request;
        System.debug(LoggingLevel.ERROR, req.requestBody.toString());
        return '';
    }
}

I used the following JavaScript in my browser to simulate the request:

(function() {
    var xhr = new XMLHttpRequest(), d = {}, i=50000;
    while(i--) d['d'+i] = i;
    d = JSON.stringify(d);
    xhr.open('POST','/services/apexrest/demoRest');
    xhr.setRequestHeader('Authorization','Bearer '+document.cookie.match(/sid=(.+?);/)[1]);
    xhr.send(d);
}())

The debug statement started off as:

{"d49999":49999,"d49998":49998,...}

Obviously, I can't post the entire debug statement here, as the payload was 727,781 characters long. My entire log was 730,252 bytes in size. Therefore, I have confirmed that the entire payload was transferring correctly, as I could visually review the string.

I then adjusted the loop from 50,000 to 200,000, resulting in a payload of 3,177,781 characters in length, and was able to confirm that the entire resource still loaded. In short, I'm not able to replicate any sort of problem with a simple 10,000 character payload, and in fact can read much larger payloads without a problem.

Finally, I adjusted the code to parse it using the standard JSON classes:

@RestResource(urlMapping='/demoRest')
global class RestDemo1 {
    @HttpPost
    global static String doPost() {
        RestRequest req = RestContext.request;
        System.debug(JSON.deserializeuntyped(req.requestbody.tostring()));
        return '';
    }
}

And was able to load a map with 50 elements that contained 1000 elements each, 678,268 characters of payload, using the following code:

(function() {
    var xhr = new XMLHttpRequest(), d = {}, i=50000, v = 1000, c = {};
    while(i--) { c[i] = i; if(i/v==Math.floor(i/v)) { d[i] = c; c = {}; } }
    d = JSON.stringify(d);
    xhr.open('POST','/services/apexrest/demoRest');
    xhr.setRequestHeader('Authorization','Bearer '+document.cookie.match(/sid=(.+?);/)[1]);
    xhr.send(d);
}())

Assuming your JSON is well-formed, your controller should load the request just fine. Since my example code is able to handle far greater numbers than what you're working with (8 times more!), the problem might be with some other error in your code.

Update 2 I wrote some new code to test this:

(function() {
    var a = new XMLHttpRequest(), b = [], c, d = 0, e;
    for(;d<10000;d++) {
        c = {}
        for(e = 0; e < 15; e++) {
            c['a'+e] = d+e;
        }
        b.push(c);
    }
    a.open('POST','/services/apexrest/phyrfox/demoRest');
    a.setRequestHeader('Content-Type','application/x-www-form-urlencoded');
    a.setRequestHeader('Authorization','Bearer '+document.cookie.match(/sid=(.+?);/)[1]);
    a.send('q='+escape(JSON.stringify(b)));
}());

I honestly expected the system to place the data into the responseBody portion of the RestRequest, so I was unpleasantly surprised when it was empty and params was populated with one element (q) that contained the JSON. I re-checked the documentation to see what it had to say about RestRequest:

Name        Return Type             Description
params      Map <String, String>    Returns the parameters that are received by the request.
requestBody Blob                    Returns or sets the body of the request.
                                    If the Apex method has no parameters, then Apex REST
                                    copies the HTTP request body into the
                                    RestRequest.requestBody property. If there are parameters,
                                    then Apex REST attempts to deserialize the data into those
                                    parameters and the data won't be deserialized into the
                                    RestRequest.requestBody property.

The requestBody bit seems to suggest that the original question is legitimate, because it would suggest that there is a bug; as far as I am concerned, I would state that the documentation or the functionality should match in order to avoid calling this a bug.

Given this, however, the params map seems to be pretty clear about what it does: it maps named parameters into the map. We would expect this behavior in other languages, such as PHP's $_REQUEST global variable, because it is documented this way. While the description for requestBody is misleading (and clearly wrong, since there are other exceptions to the rule), the fact that it is being placed in params actually does make sense.

Consider Visualforce as logical extension of this. The view state parameter is submitted with postbacks, but clearly it must be treated as a parameter instead of request-body content. The reason why the view state isn't in the URL is two-fold: first, query parameters are more exposed, both to the user and to everyone else, and query parameters are limited in size compared to a request body.

The W3C recommendations for handling POST requests suggests that all parameters should be sent with the request body instead of the URL query-string method. The logical extension of this statement is that application/x-www-form-urlencoded payloads are query strings, which in turn are query string parameters. Since the payload is considered to be part of the query string, it is no longer content, but simply an extension of the action URL's query string. This is a perfectly sane way of handling/parsing the content, and one widely accepted on the Internet as a whole.

In any case, this actually is not a bug, but standard, expected behavior consistent with W3C recommendations and implemented practices around the web with regards to POST requests. You should either change your content type, or you should use the params map.

Finally, in regards to truncation of the data, I presume that your field is configured with a maximum length of ~10k characters. You can increase this to 32,000 or so, if you'd like, but you won't be able to capture 80k worth of text in a single field. You should consider using multiple long text area fields, or even better, storing the payload as an attachment or document, which won't suffer the same limitations on storage limits.

Webhook_Test__c wht = new Webhook_Test();
insert wht;
Attachment att = new Attachment(ParentId=wht.Id, ContentType='text/plain', Body=Blob.valueOf(req.params.get('events')));
insert att;

All of this being said, I would suggest that salesforce.com update their documentation on requestBody to note that form content types will transform the payload into the query string parameters.

Related Topic