[SalesForce] the data transfer limit while using REST API in salesforce

So I think the problem is that the bulk data API is running multiple asynchronous batches and thus kicking off too many Batchable Contexts.

It looks like your batch process doesn't actually care what Id's you pass in and instead just processes everything?

If this is the case and you don't need real time (or close to realtime) updates, it might be a better idea to just schedule this to run.

Otherwise, if you need it to happen in near real time, you could use batch "chaining" in order the have the batch keep running as long as there are records to process.

The basic steps would look something like this:

1. In the finish method, add a check to see if there are more records to process. If so kick off another batch.
2. In your trigger, have it first check if the batch is already running. If so, you don't need to do anything. Otherwise, have it kick off the batch.

Update

If you have access to the JobInfo concurrencyMode param, you can set this to "Serial" and it will prevent multiple batches from running in parallel. I think this might solve you problem. If that alone doesn't work, try also reducing the batch size. Obviously, this will cause the import to take much (much) longer.

On the dataloader this Serial can be turned on by checking "Enable Serial Mode for Bulk Api".

You'll want to read Inputs and Binary File Uploads, but basically, you can't use base-64-encoded data, as far as I can tell. You could try setting Content-Transfer-Encoding to BASE64, but I'm not sure that's supported. That would look like this:

--boundary-code-1234567
Content-Disposition: ...
Content-Type: ...
Content-Transfer-Encoding: BASE64

base64ContentHere===
----boundary-code-1234567

If that doesn't work, then presumably you'll need to use binary instead. Most languages include a way to post literal binary data to a data stream, so it shouldn't be too bad. However, I've never had a need to do this, so you may need to experiment a bit.

I hope this answer helps clarify what's going on.