[SalesForce] How to publish/consume bulkified Platform Events

UPDATE

I have also discussed this topic with some more detail on on my blog here.

Based on your requirements where:

External System will push data in Salesforce.

My recommendation is to construct REST APIs for this purpose which the External Systems can utilize to send data to Salesforce. Platform Events are more targeted towards a Pub/Sub model, where you may want to "trigger" a notification from/to Salesforce to all subscribers and based on that notification the subscriber then takes necessary action. Excerpt from the documentation:

Platform events simplify the process of communicating changes and responding to them without writing complex logic. Publishers and subscribers communicate with each other through events. One or more subscribers can listen to the same event and carry out actions.

If at all you want the External System to "act upon" only when an event has occurred in Salesforce, you can then have Platform Events in this case, where the External System (the subscriber) is subscribed to the events in Salesforce and then sends data to Salesforce. Even in this case, you still need an API exposed for the External System to send data in Salesforce.

One of the most important factors to consider here is what is your integration approach. If you want a more "traditional integration", say point to point (not going into M/W details here), and that the External System always knows at what time they need to send data to Salesforce, go for REST API.

If you want your External System to notify you or if you want the External System to be notified for it to initiate the operation, that is when you should consider the event route (pub/sub).

Hope this helps.

While I haven't really been into such a design issue, but I do see a way to overcome the situation that you are facing on this part:

The 3rd party microservice listens to it and fetches that file from SF rest endpoint and then processes and updates its data back in SF

If you can break this into different pieces, you should be able to resolve this I assume. However, this is still from a very high level, you will need to definitely look towards parallel processing and deadlock scenarios and how to avoid it.

The approach I can think of is:

1. Once your microservice (MS1) fetches the event details from SF, you capture the replayId and then store that in a table in your local db or elsewhere.
2. You make the call to SF only if there does not exist the same replayId in your db
3. Let's say in the meantime another microservice (MS2) fetches the same replayId, it doesn't make call to SF as it sees the value in the db
4. In case of failures from MS1 callouts, you just delete the replayId from the db, so that any other microservice again getting that replayId can take care of making the updates.

Another approach I can think here is to utilize a Queue (say using Azure service bus or MuleSoft) along with the one mentioned above. In this approach, you can have services subscribed to the Events in Salesforce publish those events as messages in the queue. Here you will though need to make sure that the replayId being added to the Queue is unique in nature.

Now, you have another service reading from the queue which takes care of calling out to SF based on the replayId. Because you have a Queue implementation, the time you read a message from the Queue, a lock will be placed on the message and will not be available for any other service and thus you can handle any concurrency issues this way.

In summary, you will need to make sure your microservices can communicate with each other to avoid any deadlock or concurrency issues.