C program char buffer unexpected overflow

Question

I am trying to understand two different behaviors of an overflow from a C program(call it vulnerable_prog)in Linux that asks for input, in order to allow you to overflow a buffer. I understand that the compiler lays out the stack frame in particular ways, causing a bit of unpredictability sometimes. What I can't understand is the difference in the way the memory is handled when I overflow the buffer using a python script to feed 20 characters to the program, as opposed to running vulnerable_prog manually and inputting the 20 characters manually.

The example program declares an array of "char name[20]", and the goal is to overflow it and write a specific value into the other variable that will be overwritten. (This is from a classic wargaming site).

I understand that the processor(64 bit) reads 8 bytes at a time, so this requires padding of arrays that are not multiples of 8 to keep memory organized. Therefore my char [20] is actually occupying 24 bytes of memory and accessible to the processor as 8-byte words. The unexpected behavior is this:

When using a python script, the overflow behaves as follows:

$python -c'print "A"*20 + "\xre\xhe\xyt\xhe"' |  /path/vulnerable_prog

The 20 characters overflow the buffer, and the expected value is written into the correct spot in memory.

HOWEVER, when you try to overflow the buffer by running the program from the command prompt and inputting 20 characters manually, followed by the required hex string to be written to memory, you must use one additional hex character in order to have your value end up in the correct place that you want it:

$echo$ 'AAAAAAAAAAAAAAAAAAAA\xre\xhe\xyt\xhe\xaf'

(output of the 'echo' is then copied and pasted into the prompt that vulnerable_prog offers when run from the command line)

Where does this difference in the padding of the character array between the script and the command line exploitation come into play? I have been doing a lot of research of C Structure padding and reading in the ISO/IEC 9899:201x, but cannot find anything that would explain this nuance. (This is my first question on Stack Overflow so I apologize if I did not quite ask this correctly.)

Answer 1

Your Python script, when piped, actually sends 25 characters into /path/vulnerable_prog. The print statement adds a newline character. Here is your Python program plus a small Python script that counts the characters written to its standard input:

python -c'print "A"*20 + "\xre\xhe\xyt\xhe"' | python -c "import sys; print(len(sys.stdin.read()))"

I'm guessing you're not pasting the newline character that comes from echo into the program's prompt. Unfortunately, I don't think I have enough information to explain why you need 25, not 24, characters to achieve what you're attempting.

P.S. Welcome to Stack Overflow!