regular awk buffering issue when calling shell commands

Question

Some precisions:

• regular awk, NOT gawk
• AIX 6.1, and old shell: GNU bash, version 2.05b.0(1)

I am trying to display some things in the right order, hence I do pipe outputs through the shell's "sort" and "uniq"... But I can't manage to get the output in the order I write it.

Troubling code:

egrep -i "something_FREQUENT" BIGFILE | sort | awk -F',' '
      { servs=servs $1 " " ; groupe=groupe "\n   " $2 ; }
  END { print "Groupes (alphabetical order):" ;
        printf groupe "\n" | "grep . | sort | uniq | xargs echo ";
        print ; rem="extra newline to help buffering... does NOT help.";
        system(""); rem="supposed to force flush of stdout... does NOT help.";
        print "Servers:"
        printf "%s\n", servs;
      } ' # if I add a final: "| cat" here (after "'"), does NOT help?... see last example

It outputs:

Groupes (alphabetical order):

Servers:
( here a quite long list of severs... maybe until "buffer size" length? )
( etc.... but at some point :)(HERE_THE_groupe_OUTPUT)(rest of the servers here....)

??... I am at a loss : it seems that if I add : system(""); before the lines to display the servers, it does NOT help either...

Some relevant tests on the same system:

1) similar to my script: "long" output through system calls, gets garbled:

prompt$:  yes "test1" | head -10 | awk '
         { all=all "\n" $0; }
     END { print "first line:"  ;
           printf all "\n" | "sort "; 
           system(""); 
           print "last line" ;
         }'

first line:
last line

test1
test1
test1
test1
test1
test1
test1
test1
test1
test1
  # notice: the order is mixed: the shell command output is last.

2) SAME but this time we add a final '| cat' : it magically reorders the output.... but this DOES NOT HELP if I do the same in my script above...

prompt$:  yes "test2" | head -10 | awk '
         { all=all "\n" $0; }
     END { print "first line:"  ;
           printf all "\n" | "sort "; 
           system(""); 
           print "last line" ;
         }' | cat  # ONLY ADDITION: pipe awk through cat... this HELPS here (but NOT in my script!... ??)

first line:

test2
test2
test2
test2
test2
test2
test2
test2
test2
test2
last line
  # notice: the order is OK!

Answer 1

You need to close the pipe inside awk to have it flushed. For your second example do:

yes "test1" | head -10 | awk '
    { all=all "\n" $0; }
END { print "first line:"  ;
      printf all "\n" | "sort "; 
      close("sort ");
      print "last line" ;
    }'

To make it more evident in your first example, do

cmd = "grep . | sort | uniq | xargs echo ";
printf groupe "\n" | cmd;
close(cmd);