Removing NULL / empty fields

Question

just wanted to check with you could this be done better:

awk -F"\t" '{
    for (i = 1; i <= NF; i++) {
        if ($i != "NULL") {
            printf("%s%s", $i, FS);
        }
    }

    printf("\n");
}' file1

The goal is to print only non-NULL fields. For example:

echo "TestRecord001 NULL    NULL    Age 29  NULL    NULL    Name    John" | awk -F"\t" '{
    for (i = 1; i <= NF; i++) {
        if ($i != "NULL") {
            printf("%s%s", $i, FS);
        }
    }

    printf("\n");
}'

will print out: TestRecord001 Age 29 Name John

Answer 1

The same behaviour can be achieved using sed as follows:

echo -e 'TestRecord001\tNULL\tNULL\tAge\t29\tNULL\tNULL\tName\tJohn' | sed '
s/$/\t/;
s/\(\tNULL\)\+\t/\t/g;
s/^NULL\t//';

Explanation:

sed s/SearchPattern/Replacement/g. Here s indicates that replacement operation is to be done. Strings matching SearchPattern will be replaced by Replacement. g indicates that the operation will have to be performed on every match and not just on the first occurrence in a line.

1. s/$/\t/ adds a tab to the end of each line. [$ matches the end of a line]

2. \(\tNULL\)\+\t matches a string of the form \tNULL\tNULL...NULL\t. This is replaced with \t.

3. After this the only remaining NULL is the one at the beginning of a line (without \t to its left). This is matched by ^NULL\t and replaced with the empty string. [^ matches the beginning of a line]

Answer 2

Your code is good.

I can't see anything to improve upon in your original code.

awk vs other tools (e.g sed)

I think awk is a good tool for this as you are dealing with input that has clearly defined fields and field separators. The code seems more readable in awk.

Answer 3

You can print only non NULL values by using this simple awk command:

echo -e 'TestRecord001\tNULL\tNULL\tAge\t29\tNULL\tNULL\tName\tJohn' | awk -F"\t" '{gsub("NULL\t*","")}1'

gsub changes the record such that any occurence with NULL followed by a tab is deleted.