[Ethereum] FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed – JavaScript heap out of memory

javascriptweb3js

I am trying to run the follow code that queries an address to track the balance of an ERC 20 Token through the chain's. Eventually I plan on running a big array of account through all the blocks track balances for every account that has ever owned this token.

const Web3 = require('web3')
const web3 = new Web3(`http://some_url`);


const contractAddress ='some_contract_address'
var contract = new web3.eth.Contract(contractAbi, contractAddress);


const address ='0x6C1F09F6271fbE133Db38DB9c9280307F5d22160';


for (var i = 0; i < 8444182; i++) {
    contract.methods.balanceOf(address).call(null, i, (error, balance) => {

    contract.methods.decimals().call((error, decimals) => {
    //   balance = balance.div(10 ** decimals);
      console.log(balance);
    });
  })};

Unfortunately, I get the following error:

<— Last few GCs —>

[14889:0x103000000] 72963 ms: Mark-sweep 3982.1 (4188.2) -> 3971.5
(4191.2) MB, 2127.2 / 2.2 ms (average mu = 0.121, current mu = 0.031)
allocation failure scavenge might not succeed [14889:0x103000000]
75158 ms: Mark-sweep 3985.1 (4191.2) -> 3974.5 (4194.2) MB, 2133.7 /
2.3 ms (average mu = 0.076, current mu = 0.028) allocation failure scavenge might not succeed

<— JS stacktrace —>

==== JS stack trace =========================================

0: ExitFrame [pc: 0xa18d83dbe3d]
1: StubFrame [pc: 0xa18d83847bb]
2: ConstructFrame [pc: 0xa18d838d145] Security context: 0x1836dc99e6e9 <JSObject>
3: DoJoin(aka DoJoin) [0x1836dc985e91] [native array.js:~87] [pc=0xa18d863daad](this=0x1836988026f1 <undefined>,l=0x18363c5bf0a9

,m=40,A=0x1836988028c9 ,w=0x1836988029f1
,v=0x1836988029a1 )
4: Join(aka Join) [0x1836dc985ee1] [native ar…

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation
failed – JavaScript heap out of memory 1: 0x10003b59c node::Abort()
[/usr/local/opt/node@10/bin/node] 2: 0x10003b775
node::OnFatalError(char const*, char const*)
[/usr/local/opt/node@10/bin/node] 3: 0x10017bf4f
v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool)
[/usr/local/opt/node@10/bin/node] 4: 0x10017bef0
v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char
const*, bool) [/usr/local/opt/node@10/bin/node] 5: 0x100431c74
v8::internal::Heap::UpdateSurvivalStatistics(int)
[/usr/local/opt/node@10/bin/node] 6: 0x1004338b3
v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double)
[/usr/local/opt/node@10/bin/node] 7: 0x100430f4e
v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector,
v8::GCCallbackFlags) [/usr/local/opt/node@10/bin/node] 8: 0x10042fd6d
v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace,
v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)
[/usr/local/opt/node@10/bin/node] 9: 0x10043854e
v8::internal::Heap::AllocateRawWithLigthRetry(int,
v8::internal::AllocationSpace, v8::internal::AllocationAlignment)
[/usr/local/opt/node@10/bin/node] 10: 0x100438598
v8::internal::Heap::AllocateRawWithRetryOrFail(int,
v8::internal::AllocationSpace, v8::internal::AllocationAlignment)
[/usr/local/opt/node@10/bin/node] 11: 0x100418d73
v8::internal::Factory::NewFillerObject(int, bool,
v8::internal::AllocationSpace) [/usr/local/opt/node@10/bin/node] 12:
0x1005f247a v8::internal::Runtime_AllocateInNewSpace(int,
v8::internal::Object**, v8::internal::Isolate*)
[/usr/local/opt/node@10/bin/node] 13: 0xa18d83dbe3d Abort trap: 6

The error still persists after running

export NODE_OPTIONS=--max_old_space_size=4096

I will appreciate any pointers on this. Is there a more efficient way of doing this that will not draw on as much memory?

Best Answer

Change it to this:

async function run() {
    const decimals = await contract.methods.decimals().call();
    for (var i = 0; i < 8444182; i++) {
        const balance = await contract.methods.balanceOf(address).call(null, i);
        const actual = new BigNumber(balance + "e-" + decimals);
        console.log(actual.toFixed());
    }
}

run();