Execute python command and shell command (using subprocess) sequentially

Question

I am puzzled by the order of execution of python command and shell command (from subprocess).

For example, I've got a simple code:

import subprocess
import shlex

command="echo 'test'"
arg=shlex.split(command)

with open("out.txt", "w") as f:
    f.write("line1\n")
    subprocess.call(arg, stdout=f)
    f.write("line3\n")

I expect out.txt to be:

line1
test
line3

However, the actual out.txt is:

test
line1
line3

Could somebody explain the reason? Thank you.

Answer 1

Apparently, Pythons file object performs some buffering before the output is written to the underlying file descriptor (via the write() system call). So line1 ends up in the buffer but is not yet written to the file. The subprocess inherits the file descriptor which has not yet been written to, writes the line test, and then Python writes the line3 into its buffer and finally flushes line1 and line3 to the file descriptor.

To work around this, flush the file before calling the subprocess:

f.write("line1\n")
f.flush()
subprocess.call(arg, stdout=f)
f.write("line3\n")