[SalesForce] Problem loading million records in Salesforce using Bulk API

Over Twitter I was able to get an answer to question. In short, Bulk API really isn't the right framework to help with sort of problem. The Bulk API is most helpful for mass inserting/updating/deleting. It isn't really that much more efficient when querying. It would be great if the Bulk API could break a query with many results into chunks of 2000 rows. That way, the job could download all of those chunks in parallel. In theory, it's possible to do this, but you as the developer will need to determine all of the chunks (queries) and then add them all to the job. It would be amazing if the Bulk API could 'chunkify' the query for you. As of today, it cannot.

So, I found it simpler to just use the normal queryAll() calls through the WebAPI and do 10 queries at a time (using Spring Batch). The only downside is that you can only have 10 query locators open at a time.

So, I don't have a solution of how to do this with the bulk API. In fact, if you want archived Tasks it's not possible because it does NOT support the ALL ROWS keywords.

Andrew

So I think the problem is that the bulk data API is running multiple asynchronous batches and thus kicking off too many Batchable Contexts.

It looks like your batch process doesn't actually care what Id's you pass in and instead just processes everything?

If this is the case and you don't need real time (or close to realtime) updates, it might be a better idea to just schedule this to run.

Otherwise, if you need it to happen in near real time, you could use batch "chaining" in order the have the batch keep running as long as there are records to process.

The basic steps would look something like this:

1. In the finish method, add a check to see if there are more records to process. If so kick off another batch.
2. In your trigger, have it first check if the batch is already running. If so, you don't need to do anything. Otherwise, have it kick off the batch.

Update

If you have access to the JobInfo concurrencyMode param, you can set this to "Serial" and it will prevent multiple batches from running in parallel. I think this might solve you problem. If that alone doesn't work, try also reducing the batch size. Obviously, this will cause the import to take much (much) longer.

On the dataloader this Serial can be turned on by checking "Enable Serial Mode for Bulk Api".