[SalesForce] Integration approach for update and delete operation from SFDC to external system

I have a use case like this. We need to integrate external system to make outbound calls from SFDC. Lets say, like SFDC, External System is maintaining Account and Contact.

In case of update to any contact what could be the best approach. Update operation can be performed from Visualforce or standard UI. Please note, we need to make a single synchronous call to external system.

First, it will update the record in SFDC, commit the data and then make a callout?

In this case, if any error occurs from external system, we need to revert back the changes. So, possibly, we need to hold the data in some kind of map or list and revert back the changes.

First, it will make a callout and if update is successful at external system then only make an update at SFDC?

In this case, if any error occurs in SFDC, then again need to call external system to revert back the changes.

In case of deleting a contact what could be the best approach. Delete operation will be perform from salesforce standard UI.

First, it will delete the record in SFDC, commit the data and then make a callout?

In this case, if any error occurs from external system, we need to revert back the changes.
So, possibly, (a) we need undelete the record from recycle bin and re-construct the Account-Contact relationship and revert back the changes.

First, it will make a callout and if delete is successful at external system then only delete from SFDC?

Any advice, which approach to be followed. Please note that, I don't want to face this error You have uncommitted work pending. Please commit or rollback before calling out

Best Answer

Essentially you're trying to implement a two-phase commit transaction with Salesforce yet that feature as of this writing is not supported.

Your next best option is a formal integration pattern that can capture errors (e.g. in a custom object) and report / notify people as-needed when the integration fails at any point so data inconsistencies can be fixed. So you'll really be making trade-offs on which system needs to have the change occur first and rely on that response on next actions.

Salesforce Integration Design Patterns

Please see Chapter 2: Remote Process Invocation - Request and Reply of the Salesforce Integration Patterns and Practices. This talks through similar process as you've described where a Visualforce page's controller will make Http Callout to external system and receive response.

Http Callouts and Uncommitted Work

Since Http Callouts cannot be done if any DML has occurred prior to the call, you will either need to:

make the http callout before you do Salesforce DML, or
do the Salesforce DML then perform http callout asynchronously such as Queueable job or @Future method

See this help article with more details on the use case and example code.

Recommendations

In case of update to any contact what could be the best approach. Update operation can be performed from Visualforce or standard UI. Please note, we need to make a single synchronous call to external system.

Option 1: I would make the http callout, if successful, then continue with DML operation in Salesforce, else display error to user. This prevents Salesforce data from being created unless external system is successful.

Option 2: Save data to Salesforce immediately. Use near real-time integration (batchable or middleware) to query for un-synced records and post to external system keeping log of any integration errors. If errors, escalate to proper teams to investigate. Salesforce user-experience will be faster, non-blocking, and external system will eventually catch up.

In case of deleting a contact what could be the best approach. Delete operation will be perform from salesforce standard UI.

Option 1: If deletion occurs from standard delete API or standard delete button then you will only know this is happening in an apex trigger. At this point you have no opportunity to make a real-time http callout due to uncommitted work. You would have to use asynchronous apex to make http callout. The record in Salesforce would be deleted and external system may or may not be successful. You can use the undelete API method in Salesforce to restore the data if necessary, but I would worry more on why external system failed to delete and keep pushing forward, not undo it.

Option 2: Override the standard Delete action with a custom Visualforce page and Apex class. Make the http callout first, if successful then perform delete DML in apex, else display error to user. This prevents Salesforce data from being deleted unless external system is successful.

Option 3: If external system is the master in your Master Data Management scenario, then deletes in Salesforce should only ever be supported as a request to delete and not allow people to physically delete in Salesforce outside of an integration user doing the DML delete operation triggered by external system.

Personal Preference if Salesforce is "Source of Truth"

I prefer to use near real-time integrations where the external system is eventually consistent with data mastered in Salesforce.

To achieve this, I would use Batchable Apex or Middleware ETL to query for inserted/updated/deleted records using below methods then make necessary Http Callouts. Ideally, the end system could receive bulk records and perform changes all at once so as to reduce the chattiness of the integration. You would need to store in perhaps a custom setting the last timestamp when you ran these methods so that you can use it as the next start time in the next calls.

Database.getUpdated( sobjectType, startDateTime, endDateTime ) and
Database.getDeleted( sobjectType, startDateTime, endDateTime )

Apex Documentation.

SOAP API Documentation.

Also be thinking about if an error occurs and a record fails to sync then it may be left out of the next window when you call getUpdated / getDeleted. Error records may need to be re-processed in a separate batch scenario, which means you'd need a flag on the record or in some tracking object to know which records are failing and still need to be resolved.

Related Solutions

[SalesForce] how to create webhook for salesforce from external system ..

IMHO you should hit the REST API in your webhook, sending the data that you need to insert to a customObject or whatever you need to upsert.

In addition, if you need to run any logic or convert your external data, you could use a trigger before/after insert the object by REST API.

Take care with API governor limits.

[SalesForce] Best design for an external-system-to-salesforce integration

Based on what you said, all options seems viable but it's nearly impossible to tell which one is best.

Also your scenario as you tell it looks very simple. Too simple maybe. In my experience the transfer itself is easy and the tough parts are references, relationships and salesforce specials.

So I would recommend to ask yourself the following questions:

do you really have only one entity to consider?
if you say es1 is master, it doesn't look like a simple transfer but more like a sync.
if that's true, how you distinct between inserts and upserts?
Do you have an usable external unique ID?
can these records be created only in es1 or also in salesforce?
can these records be modified also in salesforce?
if one of the last 2 points is true, you could end in an 2-way-sync. Does "es1" is MASTER mean, that you overwrite all modifications in Salesforce or do you need a conflict management? If false are you aware that the records are defacto readonly in Salesforce?
do you have any relationships to other salesforce objects? That's the important thing. How will you link them?
Does es1 knows the salesforce IDs?
does salesforce know es1 IDs?
do you have any self references? (Like the parent lookup at account? That's tricky!)
do you need to worry about polymorphic relationships (like whoid or whatid on activities.)
are there any triggers OR workflow-rules OR validations rules either in salesforce or es1 which complicate things? Remember limits and errors might break bulk transactions.

a) data-loader / ETL / AppExchange

You might consider to see this pattern from a wider angle. Data-loader is only one of many external tools. ETL tools like Talend might be a great help to deal with many of the questions I raised above. The standard data-loader help could be close to zero. There are lot's of tools providing templates. Did you check AppExchange for an existing solution?

b) implement sync and transformation logic in salesforce

You end up doing the stuff above in APEX. Do you feel comfortable and do you have enough know-how plus experience in apex? Do you know the relevant limits very well? Do you have inbound limits at es1? The hourly job can be done with scheduled APEX - an example (for scheduling ONLY) looks like this:

global class TestScheduledApexFromTestMethod implements Schedulable {
    // This test runs a scheduled job at midnight Sept. 3rd. 2022
    public static String CRON_EXP = '0 0 0 3 9 ? 2022';
    global void execute(SchedulableContext ctx) {
      CronTrigger ct = [SELECT Id, CronExpression, TimesTriggered, NextFireTime
                FROM CronTrigger WHERE Id = :ctx.getTriggerId()];
      System.assertEquals(CRON_EXP, ct.CronExpression);
      System.assertEquals(0, ct.TimesTriggered);
      System.assertEquals('2022-09-03 00:00:00', String.valueOf(ct.NextFireTime));
      Account a = [SELECT Id, Name FROM Account WHERE Name = 
                  'testScheduledApexFromTestMethod'];
      a.name = 'testScheduledApexFromTestMethodUpdated';
      update a;
    }   
}

c) implement sync and logic in es1

calculate your salesforce API limits and usage-need. Does es1 provides a sufficient development framework to do the logic? Do you have the skills to do it there?

d) use Salesforce Connect

Salesforce Connect is an exciting new App Cloud integration service that empowers Salesforce users to access and manage data in external apps, whether on-premise or cloud, directly from Salesforce. With Salesforce Connect, your business can be more productive and agile, and deliver new levels of customer success.

https://trailhead.salesforce.com/de/lightning_connect/lightning_connect_setup

Salesforce Connect makes it easy to integrate two Salesforce Orgs and other systems (like SAP). However one thing prevented us from using it in most cases: the pricing. In our experience it is a couple of thousand dollars per month - often out of budget.

Conclusion

Very generally speaking in my experience and in most cases option a) will be best - but not if you limit yourself to data-loader.

Test speed and bandwidth

In all cases consider speed as a possible bottleneck. You can't guarantee that the sync completes within your 1h interval.