From How many The DAO recursive call vulnerability attacks have occurred to date?, this is what I do to get simple scripts to extract and save data:
Copy the following script into getTheDAOTransferEvents:
#!/bin/sh
# First search from 1428757 (The DAO creation) to 1736131
# First Transfer event in block 1599207
FIRSTBLOCK=${1:-1599207}
LASTBLOCK=${2:-"'latest'"}
echo "Searching for The DAO Transfer events to address 0x0000000000000000000000000000000000000000 between blocks $FIRSTBLOCK and $LASTBLOCK"
geth attach << EOF | egrep -e ",0x"
var theDAOABI = [{"anonymous":false,"inputs":[{"indexed":true,"name":"_from","type":"address"},{"indexed":true,"name":"_to","type":"address"},{"indexed":false,"name":"_amount","type":"uint256"}],"name":"Transfer","type":"event"}];
var theDAOAddress = "0xBB9bc244D798123fDe783fCc1C72d3Bb8C189413";
var theDAO = web3.eth.contract(theDAOABI).at(theDAOAddress);
var theDAOTransferEvent = theDAO.Transfer({}, {fromBlock: $FIRSTBLOCK, toBlock: $LASTBLOCK});
console.log("No,From,Block,DAOs");
var i = 0;
theDAOTransferEvent.watch(function(error, result){
var args = result.args;
if (args._to == "0x0000000000000000000000000000000000000000") {
i++;
var daos = args._amount / 1e16;
console.log(i + "," + args._from + "," + result.blockNumber + "," + daos);
}
});
theDAOTransferEvent.stopWatching();
EOF
Set the executable bit of the file using chmod 700 getTheDAOTransferEvents.
Then run the script in a separate terminal window to extract all Transfer events of interest using
./getTheDAOTransferEvents > output.txt
Q: is there a faster/better way to do what I'm trying to do (parse and extract to filesystem part of the blockchain)?
Very likely. You'll just have to test out different methods for your use cases.
Q: is there a way to use node modules and that syntax in the geth console (which is faster than --rpc)?
I've not evaluated this.
Q: is there a way to make the --rpc faster?
You can run multiple copies of the shell script at the same time, for example:
./getTheDAOTransferEvents 0 999999 > output0.txt &
./getTheDAOTransferEvents 1000000 1999999 > output1.txt &
./getTheDAOTransferEvents 2000000 2999999 > output2.txt &
./getTheDAOTransferEvents 3000000 3999999 > output3.txt &
...
Q: Initially I thought of saving to a simple.txt file but maybe is it better to store that amount of info in a db like MongoDB?
Useful if you want to access using indices. Redis is nice as well.
I like the Unix philosophy "that emphasizes building simple, short, clear, modular, and extensible code that can be easily maintained and repurposed by developers other than its creators."
I start off just creating comma (or tab) separated value files and then employ other processes to move the data into a SQL/NoSQL database.
Then if I need more speed/logic/coupling, I look to improve the process.
EDIT 23/07/2016 - Response to comments below
let me say it "geth attach << EOF | egrep -e ",0x" == MAGIC! :) that is some hardcore mambo jambo :) However I need to change the regular expression to fit my needs. I'll play with this and post a full solution. The only thing, this saves to only one file. Is there a way to save different parts of the console output to different files? - also. this doesn't give a visual feedback of the execution of the command. is there a way to show the progress? thanks!
Q: Is there a way to save different parts of the console output to different files?
I use grep to separate data sometime. Here's an example using theDAOVoter from https://github.com/bokkypoobah/TheDAOVoter to list the DAO splits.
# Generate list of DAO splits
user@Kumquat:~$ theDAOVoter --sumsplits > sumSplits
# Extract data for account 0x1368
user@Kumquat:~$ egrep "0x1368|Status" sumSplits
Prop Status Yea Nay Recipient Description
1 Expired 967598.22 4276278.60 0x13680fa2a60fd551894199f009cca20fb63a3e31
18 Expired 2200.20 3913649.01 0x13680fa2a60fd551894199f009cca20fb63a3e31
# Extract data for account 0x3d55
user@Kumquat:~$ egrep "0x3d55|Status" sumSplits
Prop Status Yea Nay Recipient Description
4 Expired 5279.34 4322941.58 0x3d5507b53d1613d8491a606ecf5c9268301095dd split
# Extract data for accounts other than 0x1368 and 0x3d55
user@Kumquat:/tmp$ egrep -v -e "0x1368|0x3d55" sumSplits
Prop Status Yea Nay Recipient Description
6 Expired 1.99 175453.91 0xbeb0b93c01297146782a5581370489a36b24deca Original intent, non-interventionist cur
7 Expired 118006.68 3967413.62 0xe82d5b10ad98d34df448b07a5a62c1affbef758f Leave me alone
8 Expired 199999.99 3931880.95 0xa72ded5c1122312d9f4ed66bf4a396139eadaf56
...
Q: this doesn't give a visual feedback of the execution of the command. is there a way to show the progress?
I use the Unix watch command to display the changes. -d highlights the differences. -n5 for example updates the display every 5 seconds.
# Watch the file size growing as the data gets written
watch -d -n5 'ls -al'
# Watch the number of lines in the file as the data gets written
watch -d -n5 'wc -l sumSplits'
# Watch the last 3 lines in the file changes as the data gets written
watch -d -n5 'tail -n3 sumSplits'
Or use tail -f sumSplits to display the file contents as the data gets written.
Best Answer
No, if they're separate transactions, it's up to the mining node's transaction ordering rules to determine if they all happen in the same block.
If you want atomic transactions, I'd recommend using a disposable, self-destructing contract to do the work for you.
Something like this:
You could probably make it even more generic, and have it accept an array of addresses and an array of data for each message, but that's a pretty complex solution for something that's probably going to require some application-specific logic anyway.