I want to load about a million records into a Data Extension from an external system. Wondering what would be the most efficient way to achieve that?
If I use SOAP/ REST API, how many records should I pass to the API call every time?
Any other suggestions?
Best Answer
Your best bet, due to the large audience is to export the records into a csv and then compress it and send it to the FTP associated with your SFMC account. You can then set up a program in automation studio that is triggered based on this file drop that decompresses it and imports it into your account. From everything I have read, this is the best/easiest way to handle massive bulk imports.
If you are determined to use API, you will likely want to do a bunch of asynchronous upserts using the REST API. I believe the limit is at around 10k per call though.
Similar questions/answers:
ExactTarget: Common data loading strategies?
Adding Bulk data to Data Extension Object in Exact target
How to bulk (import /export) to (data extensions / lists) with (soap / api / sdk)?