Contract Design – Event Logs vs Storage in Solidity: Optimization and Gas Price Considerations

I will try to anwser your question. I worked on bloom filters in cpp-ethereum and Parity.

will retrieving event logs become prohibitively slow as the blockchain becomes larger?

Not necessarily. Everything depends on the implementation, logs density (average number of logs / block) and number of cache levels.

More specifically, what is the time complexity of eth_getLogs?

In worst case, where every block contains log matching your query it is 0(n). But it's rarely a case. Bloom filters utilize probability of false positives, so the more sophisticated your filter is (more topics it has), the faster you will get your results.

Is eth_getLogs even the correct RPC request for what I'm trying to do?

Yes

To summarise, I believe, that 10s response time is caused by sub-optimal imlementation of bloom filters in go-ethereum. Here are the results of benchmarks with parity:

Find all logs from block 0 to 986082 with address: 0x33990122638b9132ca29c723bdf037f1a891a70c (should return 1602 logs).

time curl -X POST --data '{"id":8,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"0xf0be2", "address": "0x33990122638b9132ca29c723bdf037f1a891a70c"}]}' -H "Content-Type: application/json" http://127.0.0.1:3030 >> /dev/null

geth first request:

real    0m17.003s

geth second request (I assumed, that results should be cached after the first one).

real    0m18.023s

parity first request (~24x faster then geth)

real    0m0.770s

parity second request (~30x faster then geth)

real    0m0.668s

The gap between Parity and geth closes dramatically when there are no logs to be found:

Find all logs from block 0 to 986082 with address: 0x33990122638b9132ca29c723bdf037f1a891a70d (address does not exist, 0 logs returned).

time curl -X POST --data '{"id":8,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"0xf0be2", "address": "0x33990122638b9132ca29c723bdf037f1a891a70d"}]}' -H "Content-Type: application/json" http://127.0.0.1:3030 >> /dev/null

geth first request:

real    0m0.022s

geth second request

real    0m0.021s

parity first request (4x slower than geth)

real    0m0.080s

parity second request (1.5x slower than geth)

real    0m0.030s

I'll try, because bounties. :-)

Technically, no, but you could create a similar effect.

I say "no" because the blockchain is a well-ordered set of blocks each containing a well-ordered set of transactions. By extension, the blockchain is a well-ordered set of transactions starting with the first transaction in or after block 0.

Here's the kicker.

It's meant to be reviewed in order owing to the fact that a blocks after the genesis block can't be independently verified without reference to the preceeding block. In other words, to know that you are looking at the authentic 50th block (by consensus), you must have knowledge of block 49. If you start at the end you will, generally, either recursively apply logic all the way back to block 0, or place your trust in something other than your own independent assessment.

Also, you can never know you have the latest block. The most you can ever know is you have the latest block you know about (and there is probably an even newer one on the way). What, would be first on a list of events starting with the newest first? It wouldn't last very long.

To accomplish something similar to your use-case seems to imply listening to all known events and continuously inserting newer "most recent" blocks at the beginning of a list. Or (probably more efficient) append the newest to the end of your own list and then read it back in reverse when you need to. This would probably be in relation to an offchain concern (UI, cache, something else ...) - maybe "point in time" snapshots or live updates as new "most recent" log entries are revealed.

It's perfectly fair to maintain your own off-chain data sources for performance and other reasons. You can still get the benefits of blockchain when users/clients can verify the facts if they want to by confirming facts against their own copy of the chain. This is approximately what is happening when you use online block explorers and other apps that need more performance than a typical node would be capable of. You can sort that sort of thing any way you like.

Hope it helps.

UPDATE

There is a pattern to efficiently do this if you design the contract to support it. It uses very simple breadcrumbs:

pragma solidity 0.5.1;

contract ReverseOrderEvents {

    uint lastEventBlock;

    event LogEvent(address sender, uint previousEvent);

    function doSomething() public {
        emit LogEvent(msg.sender, lastEventBlock);
        lastEventBlock = block.number;
    }
}

On the client side, when lastEvent != 0 then jump to that block and "listen" for events of interest. This event will include a pointer to the block that contains the previous event. Rinse and repeat. When you hit 0 there are no previous events.

The technique is simple, and you could use separate pointers for separate logs, separate users, etc., as needed so clients can find the most important recent information quickly.

This allows a client to start with the most recent and crawl back as far as interested. The pattern can be used in stateless designs where the data is in the event log and possibly only the most recent entry is important.

Just in case it isn't clear, watch() doesn't go in reverse. To use this, you would watch() one block only (which would be fast), find what you need, and then watch() the next single block of interest, following the reverse order clues you laid down for yourself.