Python – How to Serialize Byte Array for Transaction Data in Pyethereum

pyethereumpythonraw-transaction

I've upgraded from python 2 to python 3 and some code that used to work is now broken. I can't seem to fix it. Any suggestions?

Worked in python 2

# Convert hex string to byte array
hex_string = "0x00012345...."
data_bytearray = bytearray.fromhex(hex_string.replace("0x", ""))

# Then I pass data_bytearray into transactions.Transaction and it works.  
try:
    return {'error': False, 'sign': rlp.encode(transactions.Transaction(nonce, gasPrice_int, gas_int, to, value, data_bytearray).sign(privkey)).encode('hex')}
except Exception as msg:
    return {'error': True, 'message': msg}

Same code doesn't work in python 3 and note the byte array data type has changed to 

<class 'bytearray'>   # python 3 produces this from bytearray.fromhex
<type 'bytearray'>    # python 2 produces this from bytearray.fromhex

I get the following error:

{'error': True, 'message': ObjectSerializationError('Serialization failed because of field data ("Object is not a serializable (<class \'bytearray\'>)")',)}

I tried serializing the output using pickle, but got another error:

import pickle
data_bytearray = pickle.dumps(data_bytearray)

{'error': True, 'message': AttributeError("'bytes' object has no attribute 'encode'",)}

What am I doing wrong here?

Solution:

try:
    data_hex_with0xRemoved = data_hex.replace("0x", "")
    # If I use bytearray.fromhex, it results in {'error': True, 'message': ObjectSerializationError('Serialization failed because of field data ("Object is not a serializable (<class \'bytearray\'>)")',)}
    # data = bytearray.fromhex('deadbeef')
    # So instead I use bytes.fromhex
    data = bytes.fromhex(data_hex_with0xRemoved)
    unsigned_transaction = transactions.Transaction(nonce, gasPrice_int, gas_int, to, value, data)
    raw_transaction_bytes = rlp.encode(unsigned_transaction.sign(privkey))
    raw_transaction_hex = Web3.toHex(raw_transaction_bytes)
    raw_transaction_hex_0xRemoved = raw_transaction_hex.replace("0x", "")
    return {'error': False, 'sign': raw_transaction_hex_0xRemoved}
except Exception as msg:
    return {'error': True, 'message': msg}

Best Answer

Solution

try:
    unsigned_transaction = transactions.Transaction(nonce, gasPrice_int, gas_int, to, value, data_bytearray)
    raw_transaction = rlp.encode(unsigned_transaction.sign(privkey))
    return {'error': False, 'sign': Web3.toHex(raw_transaction)}
except Exception as msg:
    return {'error': True, 'message': msg}

The only notable change is from raw_transaction.encode('hex') to Web3.toHex(raw_transaction).

The other differences from the original line were simply to reduce the amount of logic per line.

Why?

both bytes and Web3.toBytes gave me a similar error to pickle.dumps: {'error': True, 'message': AttributeError("'bytes' object has no attribute 'encode'",)}

This ^ comment was the most helpful, although it doesn't explain why you don't get that same error with bytearray (I think you should). See:

In [1]: b'\xaf'.encode('hex')
AttributeError: 'bytes' object has no attribute 'encode'

In [2]: bytearray(b'\xaf').encode('hex')
AttributeError: 'bytearray' object has no attribute 'encode'

This doesn't really have anything to do with Ethereum, only with learning how encoding and pickling works in Python 3. For example, I would expect the following to behave exactly as the first code sample from the question does:

try:
    return {'error': False, 'sign': bytearray(b'\xaf').encode('hex')}
except Exception as msg:
    return {'error': True, 'message': msg}

Encoding in Python 2 had a sloppy API, and permitted people to do nonsensical things, like "encode" binary data (which is already encoded, being binary). So, encode() was removed from the Python 3 bytes type altogether. If you want to convert bytes to hex in Python 3, there are a few ways to do it:

My personal favorite option

> Web3.toHex(b'\xaf')
'0xaf'

Alternatively, in Python 3.5+, you can use:

> b'\xaf'.hex()
'af'
# note that you don't get the "0x" prefix

Another built-in option is binascii

> import binascii
> hex_in_bytes = binascii.hexlify(b'\xaf')
b'af'

# binascii made the nonsensical choice to return the hex value in a `bytes` type
# which you can fix up by decoding to a string

> hex_in_bytes.decode('utf8')
'af'

(All of these except Web3 work with bytearray as well).

Related Topic