Using sed in shell script with multiline variables

Question

I use shell scripts to setup different types of VMs. Often these scripts include multiline variables that need to be inserted into config files at certain positions using sed.

If I create them like this, everything is fine:

VAR="Config Line1\nConfig Line2\nConfig Line 3"
sed -i "/MatchingPattern/ a $VAR" somefile

This doesn't make the script very readable though, especially since the text blocks can be quite long.

If I write them like so:

VAR="Config Line1
Config Line2
Config Line 3"
sed -i "/MatchingPattern/ a $VAR" somefile

I get an error when running the script: sed: -e expression #1, char 31: unknown command:C'`

Is there a way to use sed with variables declared like that?

Answer 1

The standard syntax for the a command is:

sed -e 'a\
first line\
  second line\
last line'

BSD (at least FreeBSD and OS/X) sed strips the leading blanks, and needs the -e to work around a bug. GNU sed allows moving the first line to right after the a\ as long as it's non-empty.

So, you'd need to preprocess the input:

VAR='content with
multiple lines
  some with lead blanks
or even backslash\'

preprocessed_VAR=$(printf '%s\n' "$VAR" |
  sed 's/\\/&&/g;s/^[[:blank:]]/\\&/;s/$/\\/')

sed -i -e "/MatchingPattern/a\\
${preprocessed_VAR%?}" somefile

(replace -i with -i '' on FreeBSD or OS/X).

On GNU/Linux and with the GNU shell (bash) or zsh, you can do instead:

sed -i '/MatchingPattern/r /dev/stdin' <<< "$VAR"

That works because bash and zsh implement here-strings with a deleted temporary files and /dev/stdin on Linux is implemented as a symlink to the file (which means it can be opened several time, and each time, it's open at the beginning).

Here you could also use GNU awk instead:

(export VAR
gawk -i /usr/share/awk/inplace.awk '{print}; /MatchingPattern/{print ENVIRON["VAR"]}' somefile)

Do not use -i inplace as gawk tries to load the inplace extension (as inplace or inplace.awk) from the current working directory first, where someone could have planted malware. The path of the inplace extension supplied with gawk may vary with the system, see the output of gawk 'BEGIN{print ENVIRON["AWKPATH"]}'

Or perl (where that -i comes from):

(export VAR
perl -pi -e '$_ .= "$ENV{VAR}\n" if /MatchingPattern/' somefile)

Answer 2

The error is due to the sed append command a needing a backslash after the actual a:

With GNU sed:

$ sed "/pattern/a\\$VAR" file.in >file.out

BSD sed, is, as far as I know, only able to insert a single line into a file with the a command, unless the newlines are properly escaped.

Closely related answer for how to solve this with BSD sed: http://unix.stackexchange.com/a/60322