Go-Ethereum – Geth-Clique: 2 Sealer POA Network Sealer Nodes Not Mining – Signed Recently, Must Wait for Others – Stuck

geth-cliquego-ethereumpoa

I have the following setup for a geth clique POA network, however blocks are not mined. The network consists of two separated nodes which are both acting as an active sealer.

Sealer 1

Start Command

geth --nodiscover --rpc --rpcaddr 0.0.0.0 --rpcport 8001 --rpccorsdomain "*" --rpcapi "personal,db,eth,net,web3,txpool,miner" --networkid 1337 --allow-insecure-unlock --unlock 0 --password <(echo -n "password") --verbosity 5 --mine --minerthreads 1

Console

DEBUG[09-20|17:00:51.199] TCP listener up                          addr=[::]:30303
INFO [09-20|17:00:51.200] New local node record                    seq=2 id=56b8ec81e5ba2619 ip=127.0.0.1 udp=0 tcp=30303
INFO [09-20|17:00:51.200] Started P2P networking                   self="enode://a476fc6ac4dc17e4d4c597fe6337a27c3ae9c7e4f3a04ab062d2a85ea8966f9d57798b09436fc91f089c1e1ba584650544811511a6b828b469b0ff4842bb4e55@127.0.0.1:30303?discport=0"
TRACE[09-20|17:00:52.022] Accepted RPC connection                  conn=@
INFO [09-20|17:00:52.167] Unlocked account                         address=0x933E73C3F959759c169eFFA4019C8fAf7D05cE33
INFO [09-20|17:00:52.168] Transaction pool price threshold updated price=1000000000
INFO [09-20|17:00:52.168] Transaction pool price threshold updated price=1000000000
INFO [09-20|17:00:52.168] Etherbase automatically configured       address=0x933E73C3F959759c169eFFA4019C8fAf7D05cE33
INFO [09-20|17:00:52.168] Commit new mining work                   number=1 sealhash=0ddf60…0e0219 uncles=0 txs=0 gas=0 fees=0 elapsed=80.755µs
TRACE[09-20|17:00:52.168] Out-of-turn signing requested            wiggle=1s
TRACE[09-20|17:00:52.168] Waiting for slot to sign and propagate   delay=-86.052ms
INFO [09-20|17:00:52.173] Successfully sealed new block            number=1 sealhash=0ddf60…0e0219 hash=0468f9…e4ef66 elapsed=5.624ms
INFO [09-20|17:00:52.173] 🔨 mined potential block                  number=1 hash=0468f9…e4ef66
DEBUG[09-20|17:00:52.173] Reinjecting stale transactions           count=0
TRACE[09-20|17:00:52.173] Propagated block                         hash=0468f9…e4ef66 recipients=0 duration=2562047h47m16.854s
TRACE[09-20|17:00:52.173] Announced block                          hash=0468f9…e4ef66 recipients=0 duration=2562047h47m16.854s
INFO [09-20|17:00:52.174] Commit new mining work                   number=2 sealhash=e3e5e5…d6c6d3 uncles=0 txs=0 gas=0 fees=0 elapsed=275.94µs
INFO [09-20|17:00:52.174] Signed recently, must wait for others

Account Status

> personal.listWallets
[{
    accounts: [{
        address: "0x933e73c3f959759c169effa4019c8faf7d05ce33",
        url: "keystore:///home/ubuntu/benchprezz/geth-clique/chainInfo/keystore/UTC--2019-09-11T14-54-07.129337000Z--933e73c3f959759c169effa4019c8faf7d05ce33"
    }],
    status: "Unlocked",
    url: "keystore:///home/ubuntu/benchprezz/geth-clique/chainInfo/keystore/UTC--2019-09-11T14-54-07.129337000Z--933e73c3f959759c169effa4019c8faf7d05ce33"
}, {
    accounts: [{
        address: "0xaf7f46a2cb4be01ec10db6c643d31fa09922ef2c",
        url: "keystore:///home/ubuntu/benchprezz/geth-clique/chainInfo/keystore/UTC--2019-09-11T14-57-21.738539000Z--af7f46a2cb4be01ec10db6c643d31fa09922ef2c"
    }],
    status: "Locked",
    url: "keystore:///home/ubuntu/benchprezz/geth-clique/chainInfo/keystore/UTC--2019-09-11T14-57-21.738539000Z--af7f46a2cb4be01ec10db6c643d31fa09922ef2c"
}]

Block Status

> blockInfo
{
  difficulty: 1,
  extraData: "0xd883010903846765746888676f312e31312e35856c696e75780000000000000044fc471624a37d0ea32a836c229bd686324869bd732b8b74b73ecfd4c16f53dd0e49bbb32dc21edec0e0819182d412f0debf24c1e8c82623587bd4f9d49f198400",
  gasLimit: 4704588,
  gasUsed: 0,
  hash: "0x8bea6bb258aa16e700d7ef23cbc66b6ed3676822b59aa78b5e462b7c5917b317",
  logsBloom: "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  miner: "0x0000000000000000000000000000000000000000",
  mixHash: "0x0000000000000000000000000000000000000000000000000000000000000000",
  nonce: "0x0000000000000000",
  number: 1,
  parentHash: "0x88cbe0e8d341d01674fce1020b7258fcdedff78d27ce288ed30e0b869d1cc496",
  receiptsRoot: "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
  sha3Uncles: "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
  size: 606,
  stateRoot: "0x133ad36f01a00cbf04031dc2fa54790748f435d2edd46cc09870923cc4299ffe",
  timestamp: 1568982903,
  totalDifficulty: 2,
  transactions: [],
  transactionsRoot: "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
  uncles: []
}

Transaction Status

> eth.pendingTransactions
[{
    blockHash: null,
    blockNumber: null,
    from: "0x933e73c3f959759c169effa4019c8faf7d05ce33",
    gas: 21000,
    gasPrice: 1000000000,
    hash: "0x558431e6168bff370fa64bc9c8387d0a7b4f04c5f4c6cdbdfbdbba0acfc82231",
    input: "0x",
    nonce: 0,
    r: "0xba0a556afe711bbc37f3cfe9f0e74d519ef0b94ac21d9fa6972f1e50e3ceed5f",
    s: "0x715cb6bde687f287a6407597b29b1f61d38e5bf428d6bdf502a6bc61fcec1b9a",
    to: "0xaf7f46a2cb4be01ec10db6c643d31fa09922ef2c",
    transactionIndex: null,
    v: "0xa95",
    value: 50000000000000000
}]

Sealer 2

Start Command

geth --nodiscover --rpc --rpcaddr 0.0.0.0 --rpcport 8002 --rpccorsdomain "*" --rpcapi "personal,db,eth,net,web3,txpool,miner" --networkid 1337 --allow-insecure-unlock --unlock 1 --password <(echo -n "password") --verbosity 5 --mine --minerthreads 1

Console

DEBUG[09-20|17:00:53.739] TCP listener up                          addr=[::]:30303
INFO [09-20|17:00:53.740] New local node record                    seq=2 id=737f8378f371b6be ip=127.0.0.1 udp=0 tcp=30303
INFO [09-20|17:00:53.740] Started P2P networking                   self="enode://d15c4f98c93beb6b444a474668ae580da0d839d8342adcd401e591ca6dbaf03f9b46b62a2d92927fe7dc4c1d30a7cd6dff29cde6f26cca7b79a025fcf45427ff@127.0.0.1:30303?discport=0"
TRACE[09-20|17:00:54.518] Accepted RPC connection                  conn=@
DEBUG[09-20|17:00:54.715] Served rpc_modules                       reqid=1 t=49.793µs
DEBUG[09-20|17:00:54.761] Served admin_addPeer                     reqid=2 t=256.732µs
TRACE[09-20|17:00:54.761] Adding static node                       node=enode://a476fc6ac4dc17e4d4c597fe6337a27c3ae9c7e4f3a04ab062d2a85ea8966f9d57798b09436fc91f089c1e1ba584650544811511a6b828b469b0ff4842bb4e55@10.195.2.37:30303
TRACE[09-20|17:00:54.761] New dial task                            task="staticdial 56b8ec81e5ba2619 10.195.2.37:30303"
DEBUG[09-20|17:00:54.786] RPC connection read error                err=EOF
DEBUG[09-20|17:00:54.804] Adding p2p peer                          id=56b8ec81e5ba2619 conn=staticdial addr=10.195.2.37:30303 peers=1 name=Geth/v1.9.3-stable-c...
TRACE[09-20|17:00:54.804] Connection set up                        id=56b8ec81e5ba2619 addr=10.195.2.37:30303 conn=staticdial inbound=false
TRACE[09-20|17:00:54.804] Dial task done                           task="staticdial 56b8ec81e5ba2619 10.195.2.37:30303"
TRACE[09-20|17:00:54.804] New dial task                            task="wait for dial hist expire (34.999998076s)"
TRACE[09-20|17:00:54.821] Starting protocol eth/63                 id=56b8ec81e5ba2619 conn=staticdial
DEBUG[09-20|17:00:54.821] Ethereum peer connected                  id=56b8ec81e5ba2619 conn=staticdial name=Geth/v1.9.3-stable-cfbb969d/linux-amd64/go1.11.5
TRACE[09-20|17:00:54.821] Registering sync peer                    peer=56b8ec81e5ba2619
TRACE[09-20|17:00:54.845] Accepted RPC connection                  conn=@
INFO [09-20|17:00:54.873] Unlocked account                         address=0xaf7f46a2CB4be01ec10Db6c643d31fa09922eF2c
INFO [09-20|17:00:54.873] Transaction pool price threshold updated price=1000000000
INFO [09-20|17:00:54.873] Transaction pool price threshold updated price=1000000000
INFO [09-20|17:00:54.873] Etherbase automatically configured       address=0x933E73C3F959759c169eFFA4019C8fAf7D05cE33
INFO [09-20|17:00:54.873] Commit new mining work                   number=1 sealhash=09a566…1df73d uncles=0 txs=0 gas=0 fees=0 elapsed=114.579µs
TRACE[09-20|17:00:54.874] Out-of-turn signing requested            wiggle=1s
WARN [09-20|17:00:54.874] Block sealing failed                     err="authentication needed: password or unlock"
DEBUG[09-20|17:00:54.945] Served rpc_modules                       reqid=1 t=28.738µs
DEBUG[09-20|17:00:54.950] Served admin_addPeer                     reqid=2 t=149.001µs
TRACE[09-20|17:00:54.951] Adding static node                       node=enode://d15c4f98c93beb6b444a474668ae580da0d839d8342adcd401e591ca6dbaf03f9b46b62a2d92927fe7dc4c1d30a7cd6dff29cde6f26cca7b79a025fcf45427ff@10.195.2.38:30303
WARN [09-20|17:00:54.951] Removing static dial candidate           id=0xa52f40         addr=10.195.2.38:30303 err="is self"
DEBUG[09-20|17:00:54.952] RPC connection read error                err=EOF
DEBUG[09-20|17:00:55.843] Couldn't add port mapping                proto=tcp extport=30303 intport=30303 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered"
INFO [09-20|17:01:03.741] Block synchronisation started 
DEBUG[09-20|17:01:03.742] Synchronising with the network           peer=56b8ec81e5ba2619 eth=63 head=0468f9…e4ef66 td=2 mode=fast
DEBUG[09-20|17:01:03.742] Retrieving remote chain height           peer=56b8ec81e5ba2619
DEBUG[09-20|17:01:03.742] Fetching batch of headers                id=56b8ec81e5ba2619 conn=staticdial count=1 fromhash=0468f9…e4ef66 skip=0 reverse=false
INFO [09-20|17:01:03.742] Mining aborted due to sync 
TRACE[09-20|17:01:03.744] Filtering headers                        peer=56b8ec81e5ba2619 headers=1
DEBUG[09-20|17:01:03.744] Remote head header identified            peer=56b8ec81e5ba2619 number=1 hash=0468f9…e4ef66
DEBUG[09-20|17:01:03.744] Looking for common ancestor              peer=56b8ec81e5ba2619 local=0 remote=1
TRACE[09-20|17:01:03.744] Span searching for common ancestor       peer=56b8ec81e5ba2619 count=2 from=0 skip=1
DEBUG[09-20|17:01:03.744] Fetching batch of headers                id=56b8ec81e5ba2619 conn=staticdial count=2 fromnum=0 skip=1 reverse=false
TRACE[09-20|17:01:03.745] Filtering headers                        peer=56b8ec81e5ba2619 headers=1
DEBUG[09-20|17:01:03.745] Found common ancestor                    peer=56b8ec81e5ba2619 number=0 hash=88cbe0…1cc496
DEBUG[09-20|17:01:03.746] Directing header downloads               peer=56b8ec81e5ba2619 origin=1
DEBUG[09-20|17:01:03.746] Downloading block bodies                 origin=1
DEBUG[09-20|17:01:03.746] Downloading transaction receipts         origin=1
TRACE[09-20|17:01:03.746] Fetching skeleton headers                peer=56b8ec81e5ba2619 count=192 from=1
DEBUG[09-20|17:01:03.746] Fetching batch of headers                id=56b8ec81e5ba2619 conn=staticdial count=128 fromnum=192 skip=191 reverse=false
TRACE[09-20|17:01:03.747] Fetching full headers                    peer=56b8ec81e5ba2619 count=192 from=1
DEBUG[09-20|17:01:03.747] Fetching batch of headers                id=56b8ec81e5ba2619 conn=staticdial count=192 fromnum=1   skip=0   reverse=false
TRACE[09-20|17:01:03.748] Filtering headers                        peer=56b8ec81e5ba2619 headers=1
TRACE[09-20|17:01:03.748] Scheduling new headers                   peer=56b8ec81e5ba2619 count=1   from=1
TRACE[09-20|17:01:03.748] Fetching full headers                    peer=56b8ec81e5ba2619 count=192 from=2
DEBUG[09-20|17:01:03.748] Fetching batch of headers                id=56b8ec81e5ba2619 conn=staticdial count=192 fromnum=2   skip=0   reverse=false
DEBUG[09-20|17:01:03.749] No more headers available                peer=56b8ec81e5ba2619
DEBUG[09-20|17:01:03.750] Header download terminated               peer=56b8ec81e5ba2619
INFO [09-20|17:01:03.751] Imported new block headers               count=1   elapsed=1.745ms   number=1 hash=0468f9…e4ef66
DEBUG[09-20|17:01:03.751] Data fetching completed                  type=receipts
DEBUG[09-20|17:01:03.751] Transaction receipt download terminated  err=nil
DEBUG[09-20|17:01:03.751] Inserting downloaded chain               items=1 firstnum=1 firsthash=0468f9…e4ef66 lastnum=1 lasthash=0468f9…e4ef66
DEBUG[09-20|17:01:03.752] Inserted new block                       number=1 hash=0468f9…e4ef66 uncles=0 txs=0 gas=0 elapsed=620.911µs root=133ad3…299ffe
INFO [09-20|17:01:03.752] Imported new chain segment               blocks=1 txs=0 mgas=0.000 elapsed=981.981µs mgasps=0.000 number=1 hash=0468f9…e4ef66 dirty=0.00B
DEBUG[09-20|17:01:03.753] Reinjecting stale transactions           count=0
DEBUG[09-20|17:01:03.846] Data fetching completed                  type=bodies
DEBUG[09-20|17:01:03.846] Block body download terminated           err=nil
DEBUG[09-20|17:01:03.846] Reset ancient limit to zero 
DEBUG[09-20|17:01:03.846] Synchronisation terminated               elapsed=104.307ms
DEBUG[09-20|17:01:03.846] Reset ancient limit to zero 
INFO [09-20|17:01:03.846] Fast sync complete, auto disabling 
TRACE[09-20|17:01:03.846] Announced block                          hash=0468f9…e4ef66 recipients=1 duration=2562047h47m16.854s
TRACE[09-20|17:01:03.847] Announced block                          id=56b8ec81e5ba2619 conn=staticdial number=1 hash=0468f9…e4ef66
INFO [09-20|17:01:03.847] Commit new mining work                   number=2 sealhash=52e874…26af49 uncles=0 txs=0 gas=0 fees=0 elapsed=208.892µs
INFO [09-20|17:01:03.847] Signed recently, must wait for others 
DEBUG[09-20|17:01:13.676] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
TRACE[09-20|17:01:29.804] Dial task done                           task="wait for dial hist expire (34.999998076s)"
DEBUG[09-20|17:01:33.677] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
DEBUG[09-20|17:01:53.573] Current full block not old enough        number=1 hash=0468f9…e4ef66 delay=90000

Account Status

> personal.listWallets
[{
    accounts: [{
        address: "0x933e73c3f959759c169effa4019c8faf7d05ce33",
        url: "keystore:///home/ubuntu/benchprezz/geth-clique/chainInfo/keystore/UTC--2019-09-11T14-54-07.129337000Z--933e73c3f959759c169effa4019c8faf7d05ce33"
    }],
    status: "Locked",
    url: "keystore:///home/ubuntu/benchprezz/geth-clique/chainInfo/keystore/UTC--2019-09-11T14-54-07.129337000Z--933e73c3f959759c169effa4019c8faf7d05ce33"
}, {
    accounts: [{
        address: "0xaf7f46a2cb4be01ec10db6c643d31fa09922ef2c",
        url: "keystore:///home/ubuntu/benchprezz/geth-clique/chainInfo/keystore/UTC--2019-09-11T14-57-21.738539000Z--af7f46a2cb4be01ec10db6c643d31fa09922ef2c"
    }],
    status: "Unlocked",
    url: "keystore:///home/ubuntu/benchprezz/geth-clique/chainInfo/keystore/UTC--2019-09-11T14-57-21.738539000Z--af7f46a2cb4be01ec10db6c643d31fa09922ef2c"
}]

Block Status

> blockInfo
{
  difficulty: 1,
  extraData: "0xd883010903846765746888676f312e31312e35856c696e75780000000000000044fc471624a37d0ea32a836c229bd686324869bd732b8b74b73ecfd4c16f53dd0e49bbb32dc21edec0e0819182d412f0debf24c1e8c82623587bd4f9d49f198400",
  gasLimit: 4704588,
  gasUsed: 0,
  hash: "0x8bea6bb258aa16e700d7ef23cbc66b6ed3676822b59aa78b5e462b7c5917b317",
  logsBloom: "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  miner: "0x0000000000000000000000000000000000000000",
  mixHash: "0x0000000000000000000000000000000000000000000000000000000000000000",
  nonce: "0x0000000000000000",
  number: 1,
  parentHash: "0x88cbe0e8d341d01674fce1020b7258fcdedff78d27ce288ed30e0b869d1cc496",
  receiptsRoot: "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
  sha3Uncles: "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
  size: 606,
  stateRoot: "0x133ad36f01a00cbf04031dc2fa54790748f435d2edd46cc09870923cc4299ffe",
  timestamp: 1568982903,
  totalDifficulty: 2,
  transactions: [],
  transactionsRoot: "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
  uncles: []
}

Transaction Status

> eth.pendingTransactions
[{
    blockHash: null,
    blockNumber: null,
    from: "0x933e73c3f959759c169effa4019c8faf7d05ce33",
    gas: 21000,
    gasPrice: 1000000000,
    hash: "0x558431e6168bff370fa64bc9c8387d0a7b4f04c5f4c6cdbdfbdbba0acfc82231",
    input: "0x",
    nonce: 0,
    r: "0xba0a556afe711bbc37f3cfe9f0e74d519ef0b94ac21d9fa6972f1e50e3ceed5f",
    s: "0x715cb6bde687f287a6407597b29b1f61d38e5bf428d6bdf502a6bc61fcec1b9a",
    to: "0xaf7f46a2cb4be01ec10db6c643d31fa09922ef2c",
    transactionIndex: null,
    v: "0xa95",
    value: 50000000000000000
}]

And finally the genesis file:

{
  "config": {
    "chainId": 1337,
    "homesteadBlock": 0,
    "eip150Block": 0,
    "eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "eip155Block": 0,
    "eip158Block": 0,
    "byzantiumBlock": 0,
    "constantinopleBlock": 0,
    "petersburgBlock": 0,
    "clique": {
      "period": 1,
      "epoch": 30000
    }
  },
  "nonce": "0x0",
  "timestamp": "0x5d790b14",
  "extraData": "0x0000000000000000000000000000000000000000000000000000000000000000933e73c3f959759c169effa4019c8faf7d05ce33af7f46a2cb4be01ec10db6c643d31fa09922ef2c0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  "gasLimit": "0x47b760",
  "difficulty": "0x1",
  "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "coinbase": "0x0000000000000000000000000000000000000000",
  "alloc": {
    "933e73c3f959759c169effa4019c8faf7d05ce33": {
      "balance": "0x200000000000000000000000000000000000000000000000000000000000000"
    },
    "af7f46a2cb4be01ec10db6c643d31fa09922ef2c": {
      "balance": "0x200000000000000000000000000000000000000000000000000000000000000"
    }
  },
  "number": "0x0",
  "gasUsed": "0x0",
  "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000"
}

Best Answer

Both sealers are configured with the same etherbase 0x933e73c3f959759c169effa4019c8faf7d05ce33.

And the second sealers fails to sign blocks because it is locked:

WARN [09-20|17:00:54.874] Block sealing failed                     err="authentication needed: password or unlock"

You have to pass the sealer address with geth parameter --miner.etherbase XXXX.

Related Topic