[SalesForce] Viable design for multiple batch jobs

I've recently run across multiple needs to execute multiple different batch jobs at the same time, and almost immediately hit the (somewhat inexplicable) SFDC limit of 5 batch jobs queued at once.

I've worked around this limit in a number of ways, none of which seem ideal to me.

  • returning a Iterable<SObject> from the start method, which contains a mix of all the SObjects I need to act upon, and then use dynamic inspection to figure out what I'm supposed to be doing on each row in the execute method. This seems fine for small jobs, but wouldn't presumably scale to millions of records due to heap size etc.
  • using a scheduled apex job that runs once, executes a batch, and then that batch in its finish method schedules another apex job (to run in say 1 minute) that executes the next batch in the chain, and so on.

Is there a more viable/flexible pattern that I could be using? What I really need is a flexible work queue where on-demand, not-very-time-sensitive batch jobs can be queued for future (serial) execution based on user activity. And that doesn't violate platform limits of batch jobs or scheduled jobs – really needs to contribute a max of 1 scheduled or batch job at any given point.

Not sure it will help, but the characteristics of the job set I'm envisioning are this:

  • each task executes on a single SObject type and does one job: e.g. setting unset fields, doing fake M-D rollups, etc.
  • some tasks are operating on a small subset of a table, others need to walk over every Contact/Lead/etc. with potentially millions of rows per batch job.
  • while some tasks are known to be needed every N hours / days, most of them are needed only after some user-driven operation: e.g. click a big red button in an admin console, do something to more than N records, or a new/updated record of a certain type requires its downstream records to be operated upon.

Best Answer

There are some tried and tested patterns for doing this, but if you'd like to save yourself some work and get extra functionality at the same time, you might want to check out Skoodat Relax. Skoodat have built a package up on the platform specifically for scheduling batches with as much power as possible and it sounds like a good match for your needs.

Otherwise, the limit of five jobs is being increased, and you can take the approach you've suggested. Depending on the exact details of the job you're running, you may be able to leverage triggers to do the processing on the other objects and just run the batch on your main record set.

Related Topic