[Ethereum] How to access Geth’s state trie

From How many The DAO recursive call vulnerability attacks have occurred to date?, this is what I do to get simple scripts to extract and save data:

Copy the following script into getTheDAOTransferEvents:

#!/bin/sh

# First search from 1428757 (The DAO creation) to 1736131
# First Transfer event in block 1599207

FIRSTBLOCK=${1:-1599207}
LASTBLOCK=${2:-"'latest'"}

echo "Searching for The DAO Transfer events to address 0x0000000000000000000000000000000000000000 between blocks $FIRSTBLOCK and $LASTBLOCK"


geth attach << EOF | egrep -e ",0x"

var theDAOABI = [{"anonymous":false,"inputs":[{"indexed":true,"name":"_from","type":"address"},{"indexed":true,"name":"_to","type":"address"},{"indexed":false,"name":"_amount","type":"uint256"}],"name":"Transfer","type":"event"}];

var theDAOAddress = "0xBB9bc244D798123fDe783fCc1C72d3Bb8C189413";

var theDAO = web3.eth.contract(theDAOABI).at(theDAOAddress);

var theDAOTransferEvent = theDAO.Transfer({}, {fromBlock: $FIRSTBLOCK, toBlock: $LASTBLOCK});

console.log("No,From,Block,DAOs");
var i = 0;
theDAOTransferEvent.watch(function(error, result){
  var args = result.args;
  if (args._to == "0x0000000000000000000000000000000000000000") {
    i++;
    var daos = args._amount / 1e16;
    console.log(i + "," + args._from + "," + result.blockNumber + "," + daos);
  }
});
theDAOTransferEvent.stopWatching();

EOF

Set the executable bit of the file using chmod 700 getTheDAOTransferEvents.

Then run the script in a separate terminal window to extract all Transfer events of interest using

./getTheDAOTransferEvents > output.txt

Q: is there a faster/better way to do what I'm trying to do (parse and extract to filesystem part of the blockchain)?

Very likely. You'll just have to test out different methods for your use cases.

Q: is there a way to use node modules and that syntax in the geth console (which is faster than --rpc)?

I've not evaluated this.

Q: is there a way to make the --rpc faster?

You can run multiple copies of the shell script at the same time, for example:

./getTheDAOTransferEvents 0 999999 > output0.txt &
./getTheDAOTransferEvents 1000000 1999999 > output1.txt &
./getTheDAOTransferEvents 2000000 2999999 > output2.txt &
./getTheDAOTransferEvents 3000000 3999999 > output3.txt &
...

Q: Initially I thought of saving to a simple.txt file but maybe is it better to store that amount of info in a db like MongoDB?

Useful if you want to access using indices. Redis is nice as well.

I like the Unix philosophy "that emphasizes building simple, short, clear, modular, and extensible code that can be easily maintained and repurposed by developers other than its creators."

I start off just creating comma (or tab) separated value files and then employ other processes to move the data into a SQL/NoSQL database.

Then if I need more speed/logic/coupling, I look to improve the process.

EDIT 23/07/2016 - Response to comments below

let me say it "geth attach << EOF | egrep -e ",0x" == MAGIC! :) that is some hardcore mambo jambo :) However I need to change the regular expression to fit my needs. I'll play with this and post a full solution. The only thing, this saves to only one file. Is there a way to save different parts of the console output to different files? - also. this doesn't give a visual feedback of the execution of the command. is there a way to show the progress? thanks!

Q: Is there a way to save different parts of the console output to different files?

I use grep to separate data sometime. Here's an example using theDAOVoter from https://github.com/bokkypoobah/TheDAOVoter to list the DAO splits.

# Generate list of DAO splits
user@Kumquat:~$ theDAOVoter --sumsplits > sumSplits

# Extract data for account 0x1368
user@Kumquat:~$ egrep "0x1368|Status" sumSplits 
       Prop Status                Yea             Nay Recipient                                  Description            
      1 Expired         967598.22      4276278.60 0x13680fa2a60fd551894199f009cca20fb63a3e31                                         
     18 Expired           2200.20      3913649.01 0x13680fa2a60fd551894199f009cca20fb63a3e31                                    

 # Extract data for account 0x3d55
user@Kumquat:~$ egrep "0x3d55|Status" sumSplits 
       Prop Status                Yea             Nay Recipient                                  Description            
      4 Expired           5279.34      4322941.58 0x3d5507b53d1613d8491a606ecf5c9268301095dd split                                   

# Extract data for accounts other than 0x1368 and 0x3d55
user@Kumquat:/tmp$ egrep -v -e "0x1368|0x3d55" sumSplits
   Prop Status                Yea             Nay Recipient                                  Description            

      6 Expired              1.99       175453.91 0xbeb0b93c01297146782a5581370489a36b24deca Original intent, non-interventionist cur

      7 Expired         118006.68      3967413.62 0xe82d5b10ad98d34df448b07a5a62c1affbef758f Leave me alone         

      8 Expired         199999.99      3931880.95 0xa72ded5c1122312d9f4ed66bf4a396139eadaf56                        

   ...

Q: this doesn't give a visual feedback of the execution of the command. is there a way to show the progress?

I use the Unix watch command to display the changes. -d highlights the differences. -n5 for example updates the display every 5 seconds.

# Watch the file size growing as the data gets written
watch -d -n5 'ls -al'

# Watch the number of lines in the file as the data gets written
watch -d -n5 'wc -l sumSplits'

# Watch the last 3 lines in the file changes as the data gets written
watch -d -n5 'tail -n3 sumSplits'

Or use tail -f sumSplits to display the file contents as the data gets written.

The light client does not download state data. It downloads the header chain, and verifies the proof-of-work but not the transactions themselves. Each block includes a state root, which allows it to verify state data against the block without having to verify the transactions. When you make an RPC call against a light client, it requests the state data from its peers. The peers provide a merkle proof, which can be verified against the state root, ensuring that the peers cannot have provided false information. But if you don't have any peers, you can't get any information from a light client.