Read file in text mode but also count raw bytes in Python?

Question

I'd like to read a file in text mode line-wise, but at the same time I'd like to insert an intermediate step which works on bytes data and basically counts the bytes read so far.

Is there a good way in the standard library to achieve that (without manually opening in bytes mode, searching for newlines, encoding, ...)? At the end I need a text reading object (being used in the CSV reader) which additionally has a byte counter.

Answer 1

Python 2

csv module works with binary files in Python 2 therefore you could just call file.tell() method to get the current byte offset in the file.

Python 3

You can't use text_file.tell() (TextIOBase instance) -- it is documented to return an opaque number that may not correspond to the actual byte position.

If it is acceptable for your use case to get the byte offset with ± bufsize precision then:

file = open(filename, 'rb') # open in binary mode
text_file = io.TextIOWrapper(file, newline='') # text mode
# pass text_file to csv module
byte_offset = file.tell() # get position ± buffering