Blockchain Fork – How to Reliably Induce a Blockchain Fork for Testing Purposes

blockchaindapp-developmentforkstestingweb3js

I would like to reliably induce a fork on a private testnet so that I can test the behavior of off-blockchain code that interacts with contracts. (see related questions as to why I would want to do that). Would probably useful for automated testing of geth and other node implementations as well.

By reliable, I mean repeatable, with the same result every time, so that I can run this in an automated test system.

A similar question appeared on the ethereum forums.

Best Answer

The Solution

  • Set up 2 connected geth instances on your private network and start them mining.
  • There is an admin.addPeer(...) command in geth to add peer nodes to the geth's list of peers, but there does not seem to be any commands to remove these peers.
  • To simulate the fork, block the network ports used by both instances of geth for their peer-to-peer connections.
  • In the Linux environment, the iptables command can be used to block the network ports. Use the equivalent firewall on Mac OSX or Windows.
  • Both disconnected geth instances would each continue mining on their individual copies of the blockchain.
  • Send some transactions separately on the two disconnected nodes and double spend the amounts at an address.
  • Reconnect the geth instances by removing the port blocking rules.
  • The double spends should disappear as the longest/highest difficulty blockchain becomes the true source and one copy of the forked blockchain is discarded.
  • This process should be repeatable, although the timings of the mining of blocks when the sent transactions are included in the separate (and eventually single) blockchain would not be the same from test to test.
  • (I'll leave the testing for another day to check how the protocol decides on which blockchain to use when the geth instances started communicating again).


The test details follows:


The Test

  • I will run two geth instances on my same computer

  • The first geth instance uses the P2P port 30301 with the following command:

    user@Kumquat:~/ForkIt$ geth --datadir ./data1           \
  --genesis ~/ForkIt/etc/CustomGenesis.json             \
  --networkid 8888 --nodiscover --mine --minerthreads 1 \
  --port 30301 --maxpeers 10 console

  • The second geth instance uses the P2P port 30302 with the following command: 

    user@Kumquat:~/ForkIt$ geth --datadir ./data2           \
  --genesis ~/ForkIt/etc/CustomGenesis.json             \
  --networkid 8888 --nodiscover --mine --minerthreads 1 \
  --port 30302 --maxpeers 10 console

  • The first geth instance uses the file ./data1/static-nodes.json to find the second geth instance. This file contains the enode information obtained using the admin.nodeInfo command from the second geth instance, where I replace the text [::] with the IP address of my computer. Here is what my ./data1/static-nodes.json looks like:

    [
  "enode://3941d48d95d4782f8b4fb7561d78642d2e53e478e5c8d3087e6e6023f5931aca3024a9679628fff775b9ddafd11d8d48f84a502fe815619be80f16b54cb1c077@192.168.1.14:30302"
]

  • The second geth instance uses the file ./data2/static-nodes.json with the enode information obtained using the admin.nodeInfo command from the first geth instance

  • To simulate the fork, I block the TCP ports 30301 and 30302 using the following commands:

    user@Kumquat:~/ForkIt$ sudo iptables -A INPUT -p tcp --dport 30301 -j DROP
user@Kumquat:~/ForkIt$ sudo iptables -A INPUT -p tcp --dport 30302 -j DROP

  • To allow the geth instances to reconnect as peers, I remove the blocking rules on TCP ports 30301 and 30302 using the following commands:

    user@Kumquat:~/ForkIt$ sudo iptables -D INPUT -p tcp --dport 30301 -j DROP
user@Kumquat:~/ForkIt$ sudo iptables -D INPUT -p tcp --dport 30302 -j DROP


The P2P Connection And Disconnection

To view the trace of the P2P connection between the geth instances, run the command admin.verbosity(6) in the geth console.

When the ports are unblocked

  • Node 1 shows the following message:

    I0412 10:11:47.379824   12467 peer.go:173] Peer 3941d48d95d4782f 192.168.1.14:30302 broadcasted 0 message(s)

  • And node 2 shows the following message:

    I0412 10:11:58.480153   12478 peer.go:173] Peer e0b2addf8107866c 192.168.1.14:35257 broadcasted 0 message(s)

When the iptables rules are created to block the P2P ports, the P2P connections between the geth instances are dropped after about 1 minute.

  • Node 1 shows the following messages:

    I0412 10:12:59.080325   12467 server.go:431] new task: static dial 3941d48d95d4782f 192.168.1.14:30302
I0412 10:12:59.080407   12467 dial.go:209] dialing enode://3941d48d95d4782f8b4fb7561d78642d2e53e478e5c8d3087e6e6023f5931aca3024a9679628fff775b9ddafd11d8d48f84a502fe815619be80f16b54cb1c077@192.168.1.14:30302
I0412 10:13:14.080977   12467 dial.go:212] dial error: dial tcp 192.168.1.14:30302: i/o timeout
> admin.peers
[]

  • And node 2 shows the following messages:

    I0412 10:20:28.784346   12478 server.go:431] new task: static dial e0b2addf8107866c 192.168.1.14:30301
I0412 10:12:58.780403   12478 dial.go:209] dialing enode://e0b2addf8107866c0e33a56f51cf800f2625ea0f4f70097ce6420b941d215c55c5404bb856d94911964865e8ac640b7ce2a2426afbf01d16d85e8f26f053a070@192.168.1.14:30301
I0412 10:13:13.780693   12478 dial.go:212] dial error: dial tcp 192.168.1.14:30301: i/o timeout
> admin.peers
[]


The Fork

When the P2P connection is blocked, the blockchain is forked with the first geth instance mining on it's separate copy of the blockchain, and the second geth instance mining on it's separate copy of the blockchain.

Below you will see the blockchain forks at block #1405. I've created a checkBlock() script that is listed at the bottom this page - the left hand column shows the block number and the right hand column shows the first 4 characters of the miner's coinbase address.

Running the command checkBlock(1401, 10000) in the first geth instance produces the following result:

1401    182434  0   Infinity    0   0   NaN     909f
1402    182523  89  364957.0    1   1   1.0     909f
1403    182612  178 273784.5    9   8   4.5     909f
1404    182523  89  243364.0    41  32  13.7    8d15
1405    182612  178 228176.0    51  10  12.8    8d15     <--- THE FORK
1406    182523  89  219045.4    69  18  13.8    8d15
1407    182612  178 212973.2    78  9   13.0    8d15
1408    182701  267 208648.6    90  12  12.9    8d15
1409    182612  178 205394.0    117 27  14.6    8d15
1410    182701  267 202872.6    127 10  14.1    8d15

Running the command checkBlock(1390, 10000) in the second geth instance produces the following results:

1401    182434  0   Infinity    0   0   NaN     909f
1402    182523  89  364957.0    1   1   1.0     909f
1403    182612  178 273784.5    9   8   4.5     909f
1404    182523  89  243364.0    41  32  13.7    8d15
1405    182612  178 228176.0    51  10  12.8    909f     <--- THE FORK
1406    182523  89  219045.4    87  36  17.4    909f
1407    182434  0   212943.5    135 48  22.5    909f
1408    182523  89  208597.7    145 10  20.7    909f
1409    182434  0   205327.2    192 47  24.0    909f
1410    182345  -89 202773.7    237 45  26.3    909f


Spending From The Same Account During The Fork

In both geth instances, I had the second account eth.accounts[1] set to the same private / public key.

Before the fork, the account balances were the same from both geth instances:

web3.fromWei(eth.getBalance(eth.accounts[1]), "ether")
10.09958

In the first geth instance after the fork, I transferred 7 ethers from eth.accounts[1] to eth.accounts[0].

> eth.sendTransaction({from: eth.accounts[1], to: eth.accounts[0], value: web3.toWei(7, "ether")})
"0xe442a4e325ff6be2fbb35ba1f381e56559df722ebc307c6ad57f3580d6b97412"
...
> web3.fromWei(eth.getBalance(eth.accounts[1]), "ether")
3.09916

In the second geth instance after the fork, I transferred 6 ethers from eth.accounts[1] to eth.accounts[0].

> eth.sendTransaction({from: eth.accounts[1], to: eth.accounts[0], value: web3.toWei(6, "ether")})
"0xa06a6a141f5ab19e0be266244eb6c07c5b9bece6dc308f5fd6244dee51606fa6"
...
> web3.fromWei(eth.getBalance(eth.accounts[1]), "ether")
4.09916

After I unblocked the P2P port, both geth instances synchronised their blockchains (presumably to the longest/highest difficulty chain) with the result that both geth instances reported a balance of 4.09916 in eth.accounts[1].


Did Difficulty Adjust Downwards When The Blockchain Forked?

Yes in the case of the second geth instance, as can be seen from block #1410 below:

> checkBlocks(1401,10000);
1401    182434  0       Infinity    0   0   NaN     909f
1402    182523  89      364957.0    1   1   1.0     909f
1403    182612  178     273784.5    9   8   4.5     909f
1404    182523  89      243364.0    41  32  13.7    8d15
1405    182612  178     228176.0    51  10  12.8    909f
1406    182523  89      219045.4    87  36  17.4    909f
1407    182434  0       212943.5    135 48  22.5    909f
1408    182523  89      208597.7    145 10  20.7    909f
1409    182434  0       205327.2    192 47  24.0    909f
1410    182345  -89     202773.7    237 45  26.3    909f
1411    182256  -178    200721.9    275 38  27.5    909f
1412    182168  -266    199035.2    306 31  27.8    909f
1413    182256  -178    197636.9    313 7   26.1    909f
1414    182168  -266    196447.0    358 45  27.5    909f
1415    182256  -178    195433.4    366 8   26.1    909f
1416    182168  -266    194549.0    388 22  25.9    909f
1417    182256  -178    193780.7    397 9   24.8    909f
1418    182344  -90     193107.9    407 10  23.9    909f
1419    182433  -1      192514.9    418 11  23.2    909f
1420    182344  -90     191979.6    465 47  24.5    909f
1421    182255  -179    191493.4    491 26  24.6    909f

I was expecting difficulty do adjust downwards because the combined blockchain had two miners, whereas the individual forked blockchain was being built by one miner. To keep the time between blocks the same (on average), difficulty would have to adjust down.


Additional Things

~/ForkIt/etc/CustomGenesis.json

{
  "alloc": {
  },
  "nonce": "0x8888888888888888",
  "difficulty": "0x020000",
  "mixhash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "coinbase": "0x8888888888888888888888888888888888888888",
  "timestamp": "0x00",
  "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "extraData": "0x",
  "gasLimit": "0x888888"
}

checkBlocks() script

For the specified range of blocks, this script will print the following information: block number, block difficulty, change in block difficulty, average block difficulty, elapsed time, change in time, average time and the first 4 characters of the miner's coinbase public key.

function checkBlocks(firstBlock, lastBlock) {
  var i;
  var firstTimestamp;
  var prevTimestamp;
  var prevDifficulty;
  var totalDifficulty = 0;
  for (i = firstBlock; i < 10000; i++) {
    var block = eth.getBlock(i);
    if (i == firstBlock) {
      firstTimestamp = block.timestamp;
      prevTimestamp = firstTimestamp;
      prevDifficulty = block.difficulty;
    }
    if (block == null)
      break;
    totalDifficulty = +totalDifficulty + +block.difficulty;
    var averageDifficulty = totalDifficulty / (i - firstBlock);
    var averageTime = (block.timestamp - firstTimestamp) / (i - firstBlock);
    console.log(block.number + "\t" + block.difficulty + 
      "\t" + (block.difficulty - prevDifficulty) + 
      "\t" + averageDifficulty.toFixed(1) + 
      "\t" + (block.timestamp - firstTimestamp) +
      "\t" + (block.timestamp - prevTimestamp) +
      "\t" + averageTime.toFixed(1) +
      "\t" + block.miner.substr(2, 4));
    prevTimestamp = block.timestamp;
  }
}

The results look like the following (the first line difference information will always be incorrect):

> checkBlocks(1401,10000);
1401    182434  0   Infinity    0   0   NaN     909f
1402    182523  89  364957.0    1   1   1.0     909f
1403    182612  178 273784.5    9   8   4.5     909f
1404    182523  89  243364.0    41  32  13.7    8d15
1405    182612  178 228176.0    51  10  12.8    8d15
1406    182523  89  219045.4    69  18  13.8    8d15
1407    182612  178 212973.2    78  9   13.0    8d15
1408    182701  267 208648.6    90  12  12.9    8d15
1409    182612  178 205394.0    117 27  14.6    8d15
Related Topic