[SalesForce] Merging records in bulk

I'm implementing Database.Merge to merge duplicate Account records in bulk, but it allows only 3 records to merge. If I perform this dml in loop I'll get Too many DML statements: 151 error, so I have created batch apex and limiting the data, I'm wondering is this the best way to do this

global class DuplicateMergeBatch implements Database.Batchable<sObject>, Database.Stateful {

public String queryString;

global Database.QueryLocator start(Database.BatchableContext BC) {
  String ph = '814606';
  queryString = 'SELECT Id, Name, Phone, Type FROM Account WHERE Phone =:ph ORDER BY CreatedDate ASC';
    return Database.getQueryLocator(queryString);
}
  global void execute(Database.BatchableContext BC, List<Account> scope) {

Map<String,List<Account>> accountMap = new Map<String,List<Account>>();
  Set<String> accountName = new Set<String>();
  Set<String> accountPhone = new Set<String>();
  Set<String> accountType = new Set<String>();

  for(Account a :scope){
     accountName.add(a.Name.toLowerCase());
     accountPhone.add(a.Phone.toLowerCase());
     accountType.add(a.Type.toLowerCase());
  }

  //Get All non formula fields of account
  String sobjectApiName = 'Account';
  Map<String, Schema.SObjectType> schemaMap = Schema.getGlobalDescribe();
  Map<String, Schema.SObjectField> fieldMap = schemaMap.get(SobjectApiName).getDescribe().fields.getMap();
  List<String> fieleList = new List<String>();
  for(String fieldName : fieldMap.keySet()){
     Schema.DescribeFieldResult fieldResult = fieldMap.get(fieldName).getDescribe();
     if(!fieldResult.isCalculated() && fieldResult.isCreateable() && !fieldResult.isExternalId()){
        fieleList.add(fieldName);
     }
     /*if(fieldName != 'isPartner' && fieldName != 'IsCustomerPortal'){
        fieleList.add(fieldName);
     }*/
  }

  String fetchFields = 'SELECT '+String.join(fieleList,',')+' FROM '+sobjectApiName+' WHERE Name IN :accountName AND Phone IN :accountPhone AND Type IN :accountType ORDER BY CreatedDate ASC limit 5';
  integer i = 1;
  for(Account a : Database.query(fetchFields)){
     String key = a.Name+a.Phone+a.Type;
     key = key.toLowerCase();
     if(accountMap.containsKey(key)){
        accountMap.get(key).add(a);
     }else {
        accountMap.put(key,new List<Account>{a});
     }
     i++;
  }
  System.debug('i >'+i);
System.debug('accountMap >> '+accountMap);
  List<Account> mergeList;
for(String key : accountMap.keySet()){
     System.debug('Size >>'+ accountMap.get(key).size());
    if(accountMap.get(key).size()>1 && accountMap.get(key).size() <6){
        mergeList = new List<Account>();
        Account masterAccount = accountMap.get(key)[0];
        mergeList.addAll(accountMap.get(key));
        mergeList.remove(0);
        System.debug('mergeList >> '+mergeList);
        for(Account a : mergeList){
            Database.MergeResult mergeResult = Database.merge(masterAccount, a, true);
        }
    }
}

//if(mergeList.size()>0)
//  update mergeList;

}

global void finish(Database.BatchableContext BC) {

}}

to execute this batch :

Database.executeBatch(new DuplicateMergeBatch(), 5);

Best Answer

This is a rather old-school method of merging duplicates. We built a better system in our org that simply queries the DuplicateRecordSet table for duplicates and merges them. Actual deduplication logic is instead made a Duplicate Rule; the system's built-in matching logic reduces the need to do anything complicated, just query and merge. It also allows us to adjust the rules if need to without a new deployment.

You have a lot of minor things wrong with your implementation, and a few big things wrong. One of the biggest errors you have here is:

 String key = a.Name+a.Phone+a.Type;

This could have some implications. Without some sort of separator, it's entirely possible (mostly through bad data entry) that you might mismatch records. For example:

Account Name:  Five5     Five
Account Phone: 55-1234   555-1234

While this is kind of silly, it can, and does happen. It may be the same account, but maybe not. I'm sure there's real-world examples where this can get you in trouble. Instead, you can use logic like I demonstrate in this answer.

Account key = new Account(
  Name=a.Name.toLowerCase(), 
  Phone=a.Phone==null? null: a.Phone.toLowerCase(), 
  Type=a.Type==null? null: a.Type.toLowerCase());

While it's a bit more verbose, it's guaranteed to keep the fields separate, regardless of the data in those fields.

You don't need to query every single field on the Account. Salesforce will only update values that are specified in the master record. Just query for the fields you're actually using.

Don't use Database.Stateful unless you have to. It's a performance hit. And no, you're not storing any state, so you don't need to use it (even queryString isn't being modified during the execute method, so does not need to be stateful).

What's the purpose of i? Maybe consider documenting this with comments...

if(accountMap.get(key).size()>1 && accountMap.get(key).size() <6){

Why 6? Your code will fail if there are more than 3 accounts to process in the same list. Also, you don't use the key in that loop. And, you copy a list for no good reason. Here's how this could be revised:

for(Account[] accList: accountMap.values()) {
    Account master = accList.remove(0);
    Account[] dupes = new Account[0];
    while(accList.size()>1) {
        dupes.add(accList.remove(0));
        if(dupes.size() == 2) {
            Database.merge(master, dupes);
            dupes.clear();
        }
    }
    if(!dupes.isEmpty()) {
        Database.merge(master, dupes);
    }
}
Related Topic