Ethereum – How to Parse Block-Level LogsBloom

bloom-filtereventsjson-rpclogs

I have been downloading blocks from EVM-based blockchains via the JSON-RPC API via eth_getBlockByNumber. In the block there is a logsBloom element which, if I understand this correctly, is an aggregate of the logs of every transaction in that block.

There are certain topics I want to search for, such as 0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef (token transfer). If I can search the block-level logsBloom filter for my topics in my software, then I can decide whether to go back to the blockchain to get any of the transaction-level logs and/or receipts.

I have limited API calls to my blockchain data provider, so I don't want to call eth_getLogs or eth_getTransactionReceipt for every single transaction on a blockchain since 99.9% of them are irrelevant to me and I simply can't send a million requests per day. I need this data historically back to genesis, so subscribing to real-time, filtered events is insufficient.

Is this possible? Can I parse and search the 256 byte block-level logsBloom data for "any transaction in the block which emitted a specific topic/event that I am asking for"?

Best Answer

To check if a topic might be possibly included in one of the transactions in the block, you must check if 3 bits in the logs bloom are set to 1.

Let's go through the example:

topic: 0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef
logsBloom: 0x0100000000000004000000000000100000000000000000000000000000200000000000000000000000000000000001000000000000000000080008000000000040000000002000000000040800000000000000000000200000000000004800000040000002000000000000000000081000000000000040000000001080020000200001000000000000000000040000004000000000000000000000000010000000008000000000000800008000000800000000008000000200000800000000000000200a000000200000000000100002004000000000000000000000000020000000000080000020000000800000000000004000000000000000080000004000

Get the keccak256 of the topic: 0xada389e1fc24a8587c776340efb91b36e675792ab631816100d55df0b5cf3cbc

You can easily check if a specific bit is set to 1 with ByteMask, e.g. 00001000 == 8. To check if a Byte (e.g. 0x33 or 51) has the bit set to 1, (51 | 8) === 51. Means when we OR all bits of the number A and number B, we should still get the A, as the same bit in A and B is 1

So from the 0xada389e1fc24a85... we have to find 3 ByteMasks and ByteIndexes to check in the bloom logs for.

We can refer to the source code, how the Bits are calculated which should be set to 1.

https://github.com/ethereum/go-ethereum/blob/793f0f9ec860f6f51e0cec943a268c10863097c7/core/types/bloom9.go#L139

And do the same thing:

function getNumber (hex: string, byteIndex: number, bytesCount: number = 1): number {
    let start = hex.startsWith('0x') ? 2 : 0;
    let i = start + byteIndex * 2;
    return parseInt(hex.substring(i, i + 2 * bytesCount), 16);
}

function getByteMasks (topic: string) {
    let hashBuf = keccak256(topic);
    let byte1Mask = 1 << (getNumber(hashBuf, 1) & 0x7);
    let byte2Mask = 1 << (getNumber(hashBuf, 3) & 0x7);
    let byte3Mask = 1 << (getNumber(hashBuf, 5) & 0x7);

    const BloomByteLength = 256;
    let byte1Pos = BloomByteLength - ((getNumber(hashBuf, 0, 2) & 0x7FF) >> 3) - 1;
    let byte2Pos = BloomByteLength - ((getNumber(hashBuf, 2, 2) & 0x7FF) >> 3) - 1;
    let byte3Pos = BloomByteLength - ((getNumber(hashBuf, 4, 2) & 0x7FF) >> 3) - 1;

    return [
        { index: byte1Pos, mask: byte1Mask },
        { index: byte2Pos, mask: byte2Mask },
        { index: byte3Pos, mask: byte3Mask },
    ];
}

And now, we can check our logsBloom if it matches the topic.

let masks = getByteMasks(topic);

let bitsAreSet = masks.every((byte, i) => {
    let b = getNumber(logsBloom, byte.index);
    let has = (b | byte.mask) === b;
    return has;
});

If bitsAreSet is false we know that none of the transactions in the block has the Transfer log. But if it is true, it doesn't guarantee that the Transfer event is present, just it could be present.

Related Topic