Queries on Large Object (>1 million rows) Fail to Execute

apexbulk-deletedeletequerysoql

We have an object with approximately 1.5 million records in Salesforce. These records are created in an external system and synced with Salesforce every 24 hours. On a 24 hour basis, we delete all 1.5 million record rows and reimport them. The problem we're having is that the object has suddenly stopped being queryable. Performing a SELECT COUNT() operation in the developer console returns the error message "No response returned from server". Performing the same query via the tooling API returns the error: "QUERY_TIMEOUT: Your query request was running for too long".

I wasn't able to come away with much from a google search – has anyone encountered this before and what was your solution? We have approximately 3 million of these records in our recycle bin and I'm wondering if that may be related to the issue…

As an alternative solution, I've written an apex queueable that deletes 10,000 rows at a time until there are no results left. Oddly enough, this executes fine until the final 10-20,000 records are queried, in which case it crashes.

Could it be that a single bad record is causing this entire issue?

Best Answer

First, a full recycle bin can cause problems, because IsDeleted is not indexed. Consider hard-deleting records as you delete them with the emptyRecycleBin function. I suspect the Queueable is probably running into similar problems because you're stacking the recycle bin full of data (it's meant to handle only a few thousand records per user).

Second, consider a smarter integration strategy. Deleting the entire database every day just to regenerate is wasteful and unnecessary. I'd recommend a mark-and-sweep type approach. Set a flag on every single record, then update all records using upsert commands (while unmarking in the process), then finally delete every marked record.

If your external system could track modifications and deletions, that'd be even better, as you won't be writing records constantly that don't need to be written.

Related Topic