Parsing Bitcoin binary data file with Python

Question

The script parses Bitcoin database (blkXXXXX.dat) files directly from raw binary to txt human readable view. And I think about how to encrease the speed of processing.

Can anyone suggest how to improve the performance? I know that the best way to encrease performance is to write this on C or C++, but what about Python? Also, is this style of programming is good?

Example cut from code (full code avaible here):

import os
import hashlib

def reverse(input):
    L = len(input)
    if (L % 2) != 0:
        return None
    else:
        Res = ''
        L = L // 2
        for i in range(L):
            T = input[i*2] + input[i*2+1]
            Res = T + Res
            T = ''
        return (Res);

f = open('blk01234.dat','rb')
tmpHex = ''
fSize = os.path.getsize(t)
while f.tell() != fSize:
    for j in range(4):
        b = f.read(1)
        b = b.encode('hex').upper()
        tmpHex = b + tmpHex
    tmpHex = ''
    for j in range(4):
        b = f.read(1)
        b = b.encode('hex').upper()
        tmpHex = b + tmpHex
    resList.append('Block size = ' + tmpHex)
    tmpHex = ''
    tmpPos3 = f.tell()
    while f.tell() != tmpPos3 + 80:
        b = f.read(1)
        b = b.encode('hex').upper()
        tmpHex = tmpHex + b
    tmpHex = tmpHex.decode('hex')
    tmpHex = hashlib.new('sha256', tmpHex).digest()
    tmpHex = hashlib.new('sha256', tmpHex).digest()
    tmpHex = tmpHex.encode('hex')
    tmpHex = tmpHex.upper()
    tmpHex = reverse(tmpHex)
    resList.append('SHA256 hash of the current block hash = ' + tmpHex)
    f.seek(tmpPos3,0)
    tmpHex = ''
    for j in range(4):
        b = f.read(1)
        b = b.encode('hex').upper()
        tmpHex = b + tmpHex
    resList.append('Version number = ' + tmpHex)
    tmpHex = ''
#.....next part of code
f.close()
f = open('blk01234.txt','w')
for j in resList:
    f.write(j + '\n')
f.close()

Answer 1

You do a lot of manipulation of bytes to construct larger structures, which you then operate on. I'd suggest looking at the standard module struct which can do a lot of this for you in a way that is much easier to grasp.

Also, if you're going to read some bytes and then read them a second time, why not just read them into memory and operate on them twice?

Finally, your output appears to be text. Instead of write() why not use print ()?

Update:

Now that I'm not on a cell phone, let me expand a bit.

First, I'll point you to this excellent answer to a question on reading 32-bit values on SO. With that as a starting point, here's a function that does the same thing (plus a helper function that gets re-used below):

import struct # Top of file

def read_buffer(binfile, size):
    """ Read in size bytes from binfile, or else!
    """
    buffer = binfile.read(size)

    if buffer is None or len(buffer) != size:
        raise EOFError

def read_uint32_le(binfile):
    """ Reads an unsigned 32-bit, little-endian integer value from binfile.
    """
    buffer = read_buffer(binfile, 4)
    u32 = struct.unpack('<I', buffer)[0] # unpack always returns tuple
    return u32

With this function, I can rewrite some of your code:

while f.tell() != fSize:
    for j in range(4):
        b = f.read(1)
        b = b.encode('hex').upper()
        tmpHex = b + tmpHex
    tmpHex = ''
    for j in range(4):
        b = f.read(1)
        b = b.encode('hex').upper()
        tmpHex = b + tmpHex
    resList.append('Block size = ' + tmpHex)

Becomes:

while f.tell() != fSize:
    ignored = read_uint32_le(f)
    blk_size = read_uint32_le(f)
    reslist.append(f"Block size = {blk_size:08X}")

And the remainder benefits greatly from the fact that there's a bytes.hex() method:

tmpHex = ''
tmpPos3 = f.tell()
while f.tell() != tmpPos3 + 80:
    b = f.read(1)
    b = b.encode('hex').upper()
    tmpHex = tmpHex + b
tmpHex = tmpHex.decode('hex')
tmpHex = hashlib.new('sha256', tmpHex).digest()
tmpHex = hashlib.new('sha256', tmpHex).digest()
tmpHex = tmpHex.encode('hex')
tmpHex = tmpHex.upper()
tmpHex = reverse(tmpHex)
resList.append('SHA256 hash of the current block hash = ' + tmpHex)
f.seek(tmpPos3,0)
tmpHex = ''
for j in range(4):
    b = f.read(1)
    b = b.encode('hex').upper()
    tmpHex = b + tmpHex
resList.append('Version number = ' + tmpHex)
tmpHex = ''

Becomes:

buffer = read_buffer(f, 30)
hash2 = sha256(sha256(buffer)).hex().upper()
reslist.append(f"SHA256 hash of the current block hash = {hash2}")
version = struct.unpack_from('<I', buffer)[0] # returns a tuple
reslist.append(f"Version number = {version:08X}")

Answer 2

Some suggestions:

• Every single line should at least be comprehensible on its own. Single letter variables destroy that comprehension, meaning that anyone reading the code needs to read the entire code and mull it over for a while to even understand what it's meant to do. Something like current_byte = file_handle.read(1) is a lot more readable than b = f.read(1).
• All the top-level code should be in either functions, methods or classes, and the file should have a main function, so that the pieces of this code can be understood and reused.
• Running this through at least one linter such as flake8 and pycodestyle will give several tips to produce more pythonic code. For example, return (Res); should be written as return res - no parentheses or semicolon, and snake_case variable names.
• There are several magic values like 80 and 4 which are hard to guess the meaning of. They should be pulled out as named variables or constants.
• Instead of closing the file handle explicitly you should use a context manager like with open(…) as file_handle:. This ensures that the file handle will be closed even if there is an exception.
• Rather than checking whether the current position in the file is at the end you should read until reading returns nothing.
• There are assignment operators such as //= which can simplify some of your expressions.
• A helpful exercise for detecting "hacky" code is to add type hints to it, and verifying it with mypy using strict settings (like not allowing Any). For example, it looks like the type of b would be Union[byte, str], which is a code smell - the variable is literally used for two different types of things.