Using binary data as a parameter in bash - any way to allow nuls?

Question

So I'd like to pass the first 512 bytes of binaryFile.dd as the second parameter to myProgram but bash strips out all the NUL chars. Is there any way to avoid this in bash or am I on a hiding to nothing?

myProgram parameter1 "$(head -c 512 binaryFile.dd)"

Answer 1

There is no way to pass a null byte in the parameter of a command. This is not because of a limitation of bash, although bash has this limitation as well. This is a limitation of the interface to run a command: it treats a null byte as the end of the parameter. There's no escaping mechanism.

Most shells don't support null bytes in variables or in the arguments of functions and builtins. Zsh is a notable exception.

$ ksh -c 'a=$(printf foo\\0bar); printf "$a"' | od -t x1
0000000 66 6f 6f
0000003
$ bash -c 'a=$(printf foo\\0bar); printf "$a"' | od -t x1
0000000 66 6f 6f 62 61 72
0000006
$ zsh -c 'a=$(printf foo\\0bar); printf "$a"' | od -t x1
0000000 66 6f 6f 00 62 61 72
0000007

But even with zsh, if you attempt to pass a parameter to an external command, then anything following a null byte is ignored — not by zsh but by the kernel.

$ zsh -c 'a=$(printf foo\\0bar); /usr/bin/printf "$a"' | od -t x1
0000000 66 6f 6f
0000003

If you want to pass null bytes to a program, you need to find some way other than a command line parameter.

head -c 512 binaryFile.dd | myProgram --read-parameter2-from-stdin parameter1
myProgram --read-parameter2-from-file=<(head -c 512 binaryFile.dd) parameter1

Answer 2

bash is not best suited to handle binary data directly.

Either use the binary data with a file, either use the hexadecimal string representing your data.

To convert in hexadecimal you can use hexdump, xxd, od.

For example to convert a 512 bytes to an hexadecimal string, use

xxd -ps -c 512 file.bin

To convert it back to binary use

echo "$myhexstring" | xxd -r -ps > file.bin

Answer 3

No, there is no way in which a string in bash could contain a NUL (\0).
Therefore, a variable (as it contains a string) could not contain a NUL.

The reason is that bash is written with the c paradigm of "a string ends in a NUL".^[1] The linux kernel also impose such limitation.^[2] But even if the kernel were to allow NULs in strings^[3] (arguments), most shells, and bash in particular, could not include NULs inside variables ^[4].

Positional parameters ($1, $2, etc.) are equivalent to variables and can also not contain NULs.

However, nuls could exist in files, in streams, and in printf:

$ printf 'test\0nuls\n' | od -vAn -tx1c
  74  65  73  74  00  6e  75  6c  73  0a
   t   e   s   t  \0   n   u   l   s  \n

As you can see, printf creates a NUL and it flows thru the pipe (|). But NULs are striped from "command executions":

$ echo $(printf 'test\0nuls\n') | od -vAn -tx1c
bash: warning: command substitution: ignored null byte in input
  74  65  73  74  6e  75  6c  73  0a
   t   e   s   t   n   u   l   s  \n

In bash 4.4 it even emits a warning. zsh in this case, silently replace the NULs by an space:

$ zsh -c ' echo $(printf "test\0nuls\n") | od -vAn -tx1c'
  74  65  73  74  20  6e  75  6c  73  0a
   t   e   s   t       n   u   l   s  \n

We can create a file that contains NULs with printf and either cat, head, tail or dd part of the file which includes the NULs:

$ printf 'test\0nuls\0in\0files\0\n' > testnul.bin
$ cat testnul.bin | xxd -ps
74657374006e756c7300696e0066696c6573000a

$ head -c 7 testnul.bin | xxd -ps
74657374006e75

$ dd if=testnul.bin bs=7 count=1 | xxd -ps
74657374006e75
1+0 records in
1+0 records out
7 bytes copied, 0.000655689 s, 10.7 kB/s

$ dd if=testnul.bin bs=7 count=1 2>/dev/null| xxd -ps
74657374006e75

In your case, there is no simple^[5] way to have the contents of a binary file as an argument. Maybe the hex representation could work:

$ myProgram "$parameter1" "$(xxd -ps -c 512 binaryFile.dd)"

---

Thanks to @Gilles for all the additional work (and detail) below.

1

^[1] All comes down to the old definition of a C string that «strings end in a NUL (\0)». This paradigm has been coded in several C libraries and tools, of which, POSIX has several examples. Like strcpy in here that states (emphasis mine):

The strcpy() function shall copy the string pointed to by s2 (including the terminating NUL character) into the array pointed to by s1.

That means that it is assumed that a string is terminated by a NUL.
Or, in other words, there could be only one NUL, the last one.

2

^[2] The execve() system call, also defined in POSIX, expects that strings (command arguments) end in a NUL. That's why even shells that could work with NULs (most don't, with the notable exception of zsh):

$ zsh -c 'a=$(printf "included\0null"); printf "$a"' | od -vAn -tx1c
  69  6e  63  6c  75  64  65  64  00  6e  75  6c  6c
   i   n   c   l   u   d   e   d  \0   n   u   l   l

Can not use NULs in arguments passed by the execve() call:

$ zsh -c 'a=$(printf "included\0null"); /usr/bin/printf "$a"' | od -vAn -tx1c
  69  6e  63  6c  75  64  65  64
   i   n   c   l   u   d   e   d

3

^[3] But even if the kernel where able to include NULs in arguments, bash will not allow them:

$ bash -c 'a=$(printf "included\0null"); /usr/bin/printf "$a"' | od -vAn -tx1c
bash: warning: command substitution: ignored null byte in input
  69  6e  63  6c  75  64  65  64  6e  75  6c  6c
   i   n   c   l   u   d   e   d   n   u   l   l

In bash 4.4 it even emits a warning when a NUL is removed.

4

^[4] Most shells, and bash in particular, could not include NULs inside variables.

$ printf 'included\0null' | od -vAn -tx1c
  69  6e  63  6c  75  64  65  64  00  6e  75  6c  6c
   i   n   c   l   u   d   e   d  \0   n   u   l   l

$ printf 'included\0null' | ( read a; printf '%s\n' "$a" | od -vAn -tx1c )
  69  6e  63  6c  75  64  65  64  6e  75  6c  6c
   i   n   c   l   u   d   e   d   n   u   l   l

If the running shell is zsh, this (instead) will work with a null:

$ zsh -c 'printf "included\0null" | ( read a; printf "%s\n" "$a" | od -vAn -tx1c )'
  69  6e  63  6c  75  64  65  64  00  6e  75  6c  6c  0a
   i   n   c   l   u   d   e   d  \0   n   u   l   l  \n

5

^[5] Meaning that a "direct" (simple) inclusion of a byte of value 0 (\0) is impossible. But an encoded (complex), either using C-string $'\0', in hex, base 64, or some equivalent, the value of zero could be included.