[Ethereum] Why is the node synchronization stuck/extremely slow at block 2,306,843

attacksgo-ethereumNetworksynchronization

Using macOS 10.12.1 with Ethereum Wallet 0.8.7, nodes are connected but Sync doesn't even start. Stuck at block 2,306,843.

What can I do? Please keep in mind, I am neither a coder nor a tech geek.

Best Answer

Update Mar 5 2017

The state cleaning was announced by Vitalik Buterin in the tweet State clearing 100% complete dated 23:07 Nov 29 2016. This time corresponds to block 2,718,436.

The Clearing Contract can be found at 0xe9c9068240d8450da314f60804debfc194b72309. There was over 10,000 transactions involved in clearing the state. The first transaction to Sweeper.sol was at block 2,675,055 in transaction 0x884d0fc7.... The last transaction to Sweeper.sol was at block 2,700,301 in transaction 0x6b651cfd....

The Sweeper.sol Contract can be found at 0xa43ebd8939d8328f5858119a3fb65f65c864c6dd. There was 35370 transactions involved in clearing the state. The first state clearing transaction was at block 2,675,055 in transaction 0x884d0fc7.... The last state clearing transaction was at block 2,717,576 in transaction 0xbf78cc00....

You can see that the daily gasUsed chart below shows the increase in gasUsed during the state clearing period:

enter image description here

One of the state clearing operations in block 2,686,351 triggered a consensus bug in geth causing a network fork in the blockchain. This was caused because the behaviour in one of the state clearing operations produced a different state compared to Parity.

Some further information in the post A state clearing FAQ.



Your blockchain is currently syncing with the block data containing the recent low gas price network attacks where the following transaction types occurred within blocks:

  • 2,283,397 to 2,301,372 - geth memory crash transaction. geth clients crashed when processing these transactions. This issue was fixed in gethFrom Shanghai, with love (1.4.12).
  • 2,283,416 to 2,379,641 - Low gas price spam transactions calling the EXTCODESIZE opcodes taking 20-60 seconds to validate due to the ~ 50,000 disk fetches needed to process each transaction.
  • 2,421,507 to 2,463,130 - Account bloat transactions where 19 million empty accounts were created. This causes the node clients to slow down as the many accounts cannot be kept in cache requiring slow re-retrieval from disk.
  • 2,468,209 to 2,474,792 - low impact transaction spam.
  • 2,550,666 to 2,551,428 - low impact transaction spam continues - from the full archive node syncing logs.
  • 2,619,660 to 2,620,384 - same as above.

Note that from #2,675,000 when the 4th hard fork occurred, the syncing slows down as the node clients are clearing out the empty accounts created by the account bloat transactions. From Is there a new DOS attack? I'm running geth 1.5.2 and after starting it his morning I'm seeing very slow block processing times and cannot catch up with the network. Worked perfectly before the hard fork. What is going on?:

This is the state clearing process working as intended. It looks like a DoS, because it's "poking" about 20 million accounts by zero-value calling each one, and each call causes the protocol to notice that the account is empty and thereby delete it from the state. The process of deleting the account from the state takes O(log(n)) disk writes, but the zero-value call only costs 700 gas, and so it these transactions take longer than usual to process.

Note that any state clearing mechanism that we could have implemented would still have required making these 20 million account deletions, so this is unfortunately an unavoidable temporary side effect; we can dial down the speed of the process if users find the current level too difficult to deal with, though that will increase the length of time the spam will have to go on.

The only way to do it is to poke empty accounts, and once all ~20 million empty accounts have been poked it will no longer be possible to do this; all modifications to the account tree will cost at least 5700 gas per modification.

Computers with hard disk drives (HDD) are slowed down more compared to computers with solid state drives (SSD). Computers with 4Gb RAM + 4Gb swap disk or less were also harder hit, with the node clients crashing.

Your options are:

  • If you are using a HDD and you can upgrade to an SSD then do so and re-sync.
  • If you have 4Gb or less RAM, consider increasing your RAM.
  • Wait until the next hard fork (in a few weeks) that aims to clean out the empty accounts. Latest update [UPDATE] Hard Fork Block Number Delay [10/31/2016].
  • Leave your computer switched on and syncing, and wait for the blocks to slowly sync.
  • Remove your old blockchain data and Ethereum Wallet will then execute a fast sync where it does not have to fully validate each of these spam transactions. See this for further information on removing your old blockchain data.
  • Use Parity as your node client with the --geth parameter that will allow the Ethereum Wallet or Mist to communicate with it. See Using Parity with Mist. Parity syncs much faster than geth and is a drop-in replacement node client for Ethereum Wallet / Mist.
  • Use Parity as your node client and it's inbuilt wallet accessed at the URL http://localhost:8080/ from your web browser. See Basic Usage.

References:



Update Nov 23 2016 - Some Results From A Full Sync

The following output from a full sync shows the speed of syncing improving after the end of the account bloat transaction attack blocks and slowing down again during the low impact transaction spam attack blocks. Further logs from the geth full node sync can be found here:

I1123 20:10:27.966288 core/blockchain.go:1042] imported 105 blocks,   846 txs ( 26.974 Mg) in    8.000s ( 3.372 Mg/s). #2463222 [c616292b… / fe945232…]
I1123 20:10:36.011591 core/blockchain.go:1042] imported 119 blocks,  1188 txs ( 34.790 Mg) in    8.045s ( 4.324 Mg/s). #2463341 [fe945232… / 64ce6572…]
I1123 20:10:44.034226 core/blockchain.go:1042] imported 188 blocks,  1394 txs ( 45.197 Mg) in    8.022s ( 5.634 Mg/s). #2463529 [64ce6572… / 2fd1bbee…]
I1123 20:10:52.045388 core/blockchain.go:1042] imported 121 blocks,  1200 txs ( 38.763 Mg) in    8.011s ( 4.839 Mg/s). #2463650 [2fd1bbee… / a65ca251…]
I1123 20:11:00.144871 core/blockchain.go:1042] imported 144 blocks,  1111 txs ( 39.831 Mg) in    8.099s ( 4.918 Mg/s). #2463794 [a65ca251… / df55fd2f…]
I1123 20:11:08.853652 core/blockchain.go:1042] imported 133 blocks,  1769 txs ( 57.122 Mg) in    8.708s ( 6.559 Mg/s). #2463927 [df55fd2f… / c2da897f…]
I1123 20:11:16.858079 core/blockchain.go:1042] imported 118 blocks,  1484 txs ( 50.059 Mg) in    8.004s ( 6.254 Mg/s). #2464045 [c2da897f… / fa0ee9b6…]
I1123 20:11:24.891666 core/blockchain.go:1042] imported 142 blocks,  1774 txs ( 59.263 Mg) in    8.033s ( 7.377 Mg/s). #2464187 [fa0ee9b6… / be4707f0…]
I1123 20:11:33.056684 core/blockchain.go:1042] imported 110 blocks,  1447 txs ( 45.366 Mg) in    8.165s ( 5.556 Mg/s). #2464297 [be4707f0… / 367c9f67…]
I1123 20:11:41.077283 core/blockchain.go:1042] imported 50 blocks,   888 txs ( 22.434 Mg) in    8.020s ( 2.797 Mg/s). #2464347 [367c9f67… / 687793e1…]
I1123 20:11:49.108136 core/blockchain.go:1042] imported 106 blocks,  1261 txs ( 33.328 Mg) in    8.030s ( 4.150 Mg/s). #2464453 [687793e1… / a7efb44e…]
I1123 20:11:57.682912 core/blockchain.go:1042] imported 120 blocks,  1182 txs ( 32.885 Mg) in    8.574s ( 3.835 Mg/s). #2464573 [a7efb44e… / 93f49e76…]
I1123 20:12:05.882500 core/blockchain.go:1042] imported 105 blocks,  1686 txs ( 51.832 Mg) in    8.199s ( 6.321 Mg/s). #2464678 [93f49e76… / 7f5245a6…]
I1123 20:12:13.888274 core/blockchain.go:1042] imported 72 blocks,  1497 txs ( 46.792 Mg) in    8.005s ( 5.845 Mg/s). #2464750 [7f5245a6… / e5b7e73c…]
I1123 20:12:22.049633 core/blockchain.go:1042] imported 74 blocks,  1838 txs ( 54.806 Mg) in    8.161s ( 6.715 Mg/s). #2464824 [e5b7e73c… / 1a2bda23…]
I1123 20:12:23.967461 core/blockchain.go:1042] imported 16 blocks,   425 txs ( 12.840 Mg) in    1.917s ( 6.695 Mg/s). #2464840 [1a2bda23… / a4d6a2bc…]
I1123 20:12:31.991847 core/blockchain.go:1042] imported 52 blocks,   834 txs ( 29.643 Mg) in    8.016s ( 3.698 Mg/s). #2464892 [12c408c2… / 4edfa863…]
I1123 20:12:39.992375 core/blockchain.go:1042] imported 109 blocks,  1341 txs ( 47.473 Mg) in    8.000s ( 5.934 Mg/s). #2465001 [4edfa863… / fa9751ef…]
I1123 20:12:48.014498 core/blockchain.go:1042] imported 164 blocks,  1453 txs ( 60.549 Mg) in    8.022s ( 7.548 Mg/s). #2465165 [fa9751ef… / c989b88e…]
I1123 20:12:56.111777 core/blockchain.go:1042] imported 122 blocks,  1591 txs ( 67.726 Mg) in    8.097s ( 8.364 Mg/s). #2465287 [c989b88e… / f61223e6…]
I1123 20:13:04.130672 core/blockchain.go:1042] imported 202 blocks,  1709 txs ( 85.518 Mg) in    8.018s (10.665 Mg/s). #2465489 [f61223e6… / 3af6790f…]
I1123 20:13:12.140467 core/blockchain.go:1042] imported 272 blocks,  1907 txs (101.256 Mg) in    8.009s (12.642 Mg/s). #2465761 [3af6790f… / 3d313f82…]
I1123 20:13:20.140735 core/blockchain.go:1042] imported 262 blocks,  2337 txs (124.650 Mg) in    8.000s (15.581 Mg/s). #2466023 [3d313f82… / 88ac88a2…]
I1123 20:13:28.425315 core/blockchain.go:1042] imported 262 blocks,  2222 txs (134.302 Mg) in    8.284s (16.211 Mg/s). #2466285 [88ac88a2… / 22ece3d2…]
I1123 20:13:36.718691 core/blockchain.go:1042] imported 73 blocks,  1780 txs ( 46.152 Mg) in    8.293s ( 5.565 Mg/s). #2466358 [22ece3d2… / df9cda17…]
I1123 20:13:44.724916 core/blockchain.go:1042] imported 176 blocks,  1659 txs ( 56.701 Mg) in    8.006s ( 7.082 Mg/s). #2466534 [df9cda17… / cdd18871…]
I1123 20:13:52.735020 core/blockchain.go:1042] imported 351 blocks,  1896 txs ( 71.993 Mg) in    8.010s ( 8.988 Mg/s). #2466885 [cdd18871… / dfa67df6…]
I1123 20:13:52.879160 core/blockchain.go:1042] imported 3 blocks,    64 txs (  0.525 Mg) in 144.144ms ( 3.642 Mg/s). #2466888 [dfa67df6… / 87ca9736…]
I1123 20:14:01.017302 core/blockchain.go:1042] imported 186 blocks,  1584 txs ( 48.823 Mg) in    8.131s ( 6.004 Mg/s). #2467074 [e13b5025… / fb837146…]
I1123 20:14:09.050383 core/blockchain.go:1042] imported 35 blocks,  1476 txs ( 34.768 Mg) in    8.033s ( 4.328 Mg/s). #2467109 [fb837146… / 94b49f56…]
I1123 20:14:17.065023 core/blockchain.go:1042] imported 345 blocks,  1688 txs ( 63.117 Mg) in    8.014s ( 7.875 Mg/s). #2467454 [94b49f56… / 48f9ac8e…]
I1123 20:14:25.157649 core/blockchain.go:1042] imported 372 blocks,  1478 txs ( 75.912 Mg) in    8.092s ( 9.380 Mg/s). #2467826 [48f9ac8e… / 1d0a3170…]
I1123 20:14:33.531035 core/blockchain.go:1042] imported 383 blocks,  1512 txs ( 90.980 Mg) in    8.373s (10.865 Mg/s). #2468209 [1d0a3170… / 0810e44e…]
I1123 20:14:42.139109 core/blockchain.go:1042] imported 71 blocks,   295 txs ( 38.829 Mg) in    8.608s ( 4.511 Mg/s). #2468280 [0810e44e… / 48dfb626…]
I1123 20:14:50.643369 core/blockchain.go:1042] imported 18 blocks,   178 txs ( 32.282 Mg) in    8.504s ( 3.796 Mg/s). #2468298 [48dfb626… / cf9a853c…]
I1123 20:14:58.677977 core/blockchain.go:1042] imported 21 blocks,   366 txs ( 33.985 Mg) in    8.034s ( 4.230 Mg/s). #2468319 [cf9a853c… / a2f2c832…]
I1123 20:15:07.211879 core/blockchain.go:1042] imported 25 blocks,   321 txs ( 31.820 Mg) in    8.533s ( 3.729 Mg/s). #2468344 [a2f2c832… / d40e6bff…]
I1123 20:15:15.291535 core/blockchain.go:1042] imported 40 blocks,   199 txs ( 32.407 Mg) in    8.079s ( 4.011 Mg/s). #2468384 [d40e6bff… / afb38f04…]
I1123 20:15:23.888700 core/blockchain.go:1042] imported 32 blocks,   241 txs ( 32.785 Mg) in    8.597s ( 3.813 Mg/s). #2468416 [afb38f04… / 777691ae…]
I1123 20:15:32.611002 core/blockchain.go:1042] imported 39 blocks,   229 txs ( 35.176 Mg) in    8.722s ( 4.033 Mg/s). #2468455 [777691ae… / 0233d4a9…]
I1123 20:15:40.999610 core/blockchain.go:1042] imported 32 blocks,   215 txs ( 31.990 Mg) in    8.388s ( 3.813 Mg/s). #2468487 [0233d4a9… / a5c0b3f5…]
I1123 20:15:49.003363 core/blockchain.go:1042] imported 23 blocks,   146 txs ( 27.733 Mg) in    8.003s ( 3.465 Mg/s). #2468510 [a5c0b3f5… / eee46cd5…]
I1123 20:15:58.055329 core/blockchain.go:1042] imported 25 blocks,   140 txs ( 30.918 Mg) in    9.051s ( 3.416 Mg/s). #2468535 [eee46cd5… / b602d4fd…]
I1123 20:16:06.683331 core/blockchain.go:1042] imported 20 blocks,   142 txs ( 32.333 Mg) in    8.628s ( 3.748 Mg/s). #2468555 [b602d4fd… / 3e227bc8…]
I1123 20:16:15.561694 core/blockchain.go:1042] imported 28 blocks,   203 txs ( 31.675 Mg) in    8.878s ( 3.568 Mg/s). #2468583 [3e227bc8… / 3d69d1f4…]
I1123 20:16:23.703654 core/blockchain.go:1042] imported 13 blocks,   155 txs ( 25.326 Mg) in    8.141s ( 3.111 Mg/s). #2468596 [3d69d1f4… / 5123b504…]
I1123 20:16:31.789138 core/blockchain.go:1042] imported 24 blocks,   181 txs ( 33.079 Mg) in    8.085s ( 4.091 Mg/s). #2468620 [5123b504… / 02a2ba0b…]
I1123 20:16:40.192777 core/blockchain.go:1042] imported 38 blocks,   210 txs ( 34.546 Mg) in    8.403s ( 4.111 Mg/s). #2468658 [02a2ba0b… / 7974dacd…]

The following output shows the syncing speed improving after the low impact transaction spam attack blocks:

I1124 01:04:19.068991 core/blockchain.go:1042] imported 5 blocks,    52 txs (  2.233 Mg) in    8.827s ( 0.253 Mg/s). #2474655 [e6538055… / eea6dea1…]
I1124 01:04:27.308275 core/blockchain.go:1042] imported 1 blocks,    15 txs (  1.476 Mg) in    8.239s ( 0.179 Mg/s). #2474656 [a8f76648…]
I1124 01:04:44.682904 core/blockchain.go:1042] imported 4 blocks,    40 txs (  1.947 Mg) in   17.374s ( 0.112 Mg/s). #2474660 [a8f76648… / d0b6d7a1…]
I1124 01:05:18.012660 core/blockchain.go:1042] imported 1 blocks,    29 txs (  1.568 Mg) in   33.329s ( 0.047 Mg/s). #2474661 [e408f930…]
I1124 01:05:50.303563 core/blockchain.go:1042] imported 23 blocks,   222 txs (  5.848 Mg) in   32.290s ( 0.181 Mg/s). #2474684 [e408f930… / 45a21a9a…]
I1124 01:05:58.533517 core/blockchain.go:1042] imported 39 blocks,   533 txs ( 13.108 Mg) in    8.229s ( 1.593 Mg/s). #2474723 [45a21a9a… / 3fbc7621…]
I1124 01:06:08.073414 core/blockchain.go:1042] imported 3 blocks,    53 txs (  1.898 Mg) in    9.539s ( 0.199 Mg/s). #2474726 [3fbc7621… / a496ad25…]
I1124 01:06:22.011190 core/blockchain.go:1042] imported 4 blocks,    35 txs (  2.834 Mg) in   13.937s ( 0.203 Mg/s). #2474730 [a496ad25… / 09b622ec…]
I1124 01:06:30.460992 core/blockchain.go:1042] imported 5 blocks,    56 txs (  2.821 Mg) in    8.449s ( 0.334 Mg/s). #2474735 [09b622ec… / b627e318…]
I1124 01:06:39.170590 core/blockchain.go:1042] imported 13 blocks,    71 txs (  2.581 Mg) in    8.709s ( 0.296 Mg/s). #2474748 [b627e318… / f28519bd…]
I1124 01:06:51.495202 core/blockchain.go:1042] imported 17 blocks,   240 txs (  7.728 Mg) in   12.324s ( 0.627 Mg/s). #2474765 [f28519bd… / db2834dc…]
I1124 01:07:01.042784 core/blockchain.go:1042] imported 20 blocks,   213 txs (  6.948 Mg) in    9.547s ( 0.728 Mg/s). #2474785 [db2834dc… / 1cfa931e…]
I1124 01:07:13.004212 core/blockchain.go:1042] imported 5 blocks,    33 txs (  3.086 Mg) in   11.961s ( 0.258 Mg/s). #2474790 [1cfa931e… / a082654e…]
I1124 01:07:21.120528 core/blockchain.go:1042] imported 159 blocks,  1346 txs ( 40.776 Mg) in    8.116s ( 5.024 Mg/s). #2474949 [a082654e… / ab0ff686…]
I1124 01:07:28.858860 core/blockchain.go:1042] imported 131 blocks,  1244 txs ( 39.997 Mg) in    7.738s ( 5.169 Mg/s). #2475080 [ab0ff686… / cb13b02f…]
I1124 01:07:37.085006 core/blockchain.go:1042] imported 120 blocks,  1186 txs ( 31.406 Mg) in    8.160s ( 3.848 Mg/s). #2475200 [7187e832… / 7ac46f03…]
I1124 01:07:45.103519 core/blockchain.go:1042] imported 142 blocks,  1697 txs ( 42.202 Mg) in    8.018s ( 5.263 Mg/s). #2475342 [7ac46f03… / 5d5c85da…]
I1124 01:07:53.420140 core/blockchain.go:1042] imported 263 blocks,  1411 txs ( 38.311 Mg) in    8.316s ( 4.607 Mg/s). #2475605 [5d5c85da… / 448dc356…]
I1124 01:08:01.424666 core/blockchain.go:1042] imported 239 blocks,  2191 txs ( 60.084 Mg) in    8.004s ( 7.506 Mg/s). #2475844 [448dc356… / b1577b08…]
I1124 01:08:09.506521 core/blockchain.go:1042] imported 375 blocks,  1999 txs ( 53.716 Mg) in    8.081s ( 6.646 Mg/s). #2476219 [b1577b08… / 392ae03e…]
I1124 01:08:17.710115 core/blockchain.go:1042] imported 74 blocks,  1736 txs ( 39.324 Mg) in    8.203s ( 4.793 Mg/s). #2476293 [392ae03e… / f635d6ea…]
I1124 01:08:25.725640 core/blockchain.go:1042] imported 156 blocks,  2193 txs ( 52.269 Mg) in    8.015s ( 6.521 Mg/s). #2476449 [f635d6ea… / c344899d…]
Related Topic