[SalesForce] Reliable SOAP webservice callout

Problem

I have to build a reliable integration. This means that external callouts should be delivered even if the remote SOAP webservice is down. (when the ws is online)

The idea is to keep the callouts messages if the webservice is down and will retry to send them after some time.

Background

Thinking on the solution I noticed some important facts:

  • The order of sent messages should be respected
  • There are more than one external method
  • The sent record data should be the current data, not the data when the callout was trigger.
  • I have a dummy remote method to check the WS availability
  • The response should be asynchronous (CallBack WS)
  • I can't use outbound messages

Approach

My approach is to insert the message to send into a queue and consume it from a scheduled job.
The stored message will have:

  • recordId in order to get the current data
  • message type in order to know which remote method will be called
  • extra data (JSON serialized) to add some useful context
  • timestamp

When the cron is executed I planning something like this: 

kill schedule  // to avoid more than one consumer
msgs = [select from queue order by timestamp asc limit 200] //limit to ensure bulk-safe 
if !msgs.empty
   if WS.isAvailable // check the remote webserver
      for(m:msgs)
         switch m.type
            case typeA
                 ws = new WSA(m.id,m.extraData); //instance the WS wrapper
            case....

         ws.send(m); // I'll explain this later
         consumed.add(m); // to remove later on queue

      delete consumed
schedule job

WSA, WSB, .. will extends from WS

class WS{

  abstract getData(); // this will retrive the needed data using the record Id
  abstract callRemote(); // this will instance and call the remote ws

  public static send(){
     this.getData();
     this.callRemote();
  }

}

This is a draw to illustrate the idea

enter image description here

Questions

  1. Can you see any problem on my approach?
  2. Have this approach any governor limitation? It is planned for Unlimited
  3. Is out there any better/simpler approach to build a 99.9% reliable integration (SF > WS SOAP) ?
  4. Could you suggest any improvement?

Update

I've discovered a possible problem: if queue.size() > 200 (I put that limit for soql limitation during the process) in the scheduled time. The queue keeps growing and never will be totally consumed.

How could I solve this?

Best Answer

You can achieve the vision in the drawing. As I see it, you need the following pieces:

  • An abstract base class (WS) that the other classes will inherit from, or an interface, depending on the exact use of the other classes.
  • A scheduled class that executes periodically to check if the server is available, and if so, kick off a batch process.
  • A batch class (can be the same as the prior class-- they're just interfaces) that queries pending messages, and attempts to execute them; once completed, they can be marked as such.
  • A custom object that holds the queued data, which you can sort by creation date in order to guarantee transaction order.

There's no reason why the system infrastructure couldn't support this design model. It has an added level of robustness, because the normal Outbound Messaging system will discard messages that exceed a certain time limit (24 hours), and does not guarantee message order.

Can you see any problem on my approach?

It is ideally achievable, and probably easier to build than you realize.

Have this approach any governor limitation? It is planned for Unlimited

Batch methods only get to use 10 callouts per transaction, so you'll have to make sure you are setting the scope size appropriately. I'd probably save a "connection check" and simply try to execute the function-- if it fails, you can catch the exception and retry those messages later.

Is out there any better/simpler approach to build a 99.9% reliable integration (SF > WS SOAP) ?

In theory, a polling loop on the other server might be more efficient in the long term, but the salesforce.com infrastructure can easily support this logic for thousands or even tens of thousands of calls per day (or more). The guaranteed transaction order piece is what makes this proposed solution attractive.

Could you suggest any improvement?

The system should work perfectly well. The tricky part might be the actual implementation, but this is a sound design.

Related Topic