Split text file into lines with fixed number of words

Use xargs (17 seconds):

xargs -n1000 <file >output

It uses the -n flag of xargs which defines the max number of arguments. Just change 1000 to 500 or whatever limit you want.

I made a test file with 10^7 words:

$ wc -w file
10000000 file

Here are the time stats:

$ time xargs -n1000 <file >output
real    0m16.677s
user    0m1.084s
sys     0m0.744s

Perl seems quite astonishingly good at this:

Create a file with 10,000,000 space separated words

for ((i=1; i<=10000000; i++)); do printf "%s " $RANDOM ; done > one.line

Now, perl to add a newline after each 1,000 words

time perl -pe '
    s{ 
        (?:\S+\s+){999} \S+   # 1000 words
        \K                    # then reset start of match
        \s+                   # and the next bit of whitespace
    }
    {\n}gx                    # replace whitespace with newline
' one.line > many.line

Timing

real    0m1.074s
user    0m0.996s
sys     0m0.076s

verify results

$ wc one.line many.line
        0  10000000  56608931 one.line
    10000  10000000  56608931 many.line
    10000  20000000 113217862 total

The accepted awk solution took just over 5 sec on my input file.

The venerable fmt(1) command, while not strictly operating on "a particular number of words" can fairly quickly wrap long lines to a particular goal (or maximum) width:

perl -e 'for (1..100) { print "a"x int 3+rand(7), " " }' | fmt

Or with modern perl, for a specific number of words, say, 10, and assuming a single space as the word boundary:

... | perl -ple 's/(.*? ){10}\K/\n/g'

The same sed command can be simplified by specifying how many word-space patterns you want to match. I didn't have any big string files to test it out on, but without the loops in your original script this should run as fast as your processor can stream the data. Added benefit, it'll work equally well on multi-line files.

n=500; sed -r "s/((\w+\s){$n})/\1\n/g" <input.txt >output.txt

Not really suitable when Number of words is a big number but if it's a small number (and ideally, no leading/trailing spaces in your one-line file) this should be quite fast (e.g. 5 words per line):

tr -s '[[:blank:]]' '\n' <input.txt | paste -d' ' - - - - - >output.txt

wonder how many msec this takes in go, just made a prototype didnt compile

//wordsplit.go

// go build wordsplit.go && ./wordsplit bigtext.txt

package main


import (
    "fmt"
    "io/ioutil"
    "log"
    "os"
    "strings"
)


func main() {
    myfile, err := os.Open(os.Args[0])
    if err != nil {
        log.Fatal(err)
    }
    defer myfile.Close()
    data, err := ioutil.ReadAll()
    if err != nil {
        log.Fatal(err)
    }
    words := strings.Split(data, " ")
    newfile, err := os.Create("output.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer newfile.close
    for i := 0; i < len(words)-10; i+10 {
        newfile.WriteString(words[i:i+10])
    }
    newfile.WriteString(words[-(len(words)%10):])
    fmt.Printf("Formatted %s into 10 word lines in output.txt", os.Args[0])
}

The coreutils pr command is another candidate: the only wrinkle seems to be that it is necessary to force the page width to be large enough to accommodate the output width.

Using a file created using @Glenn_Jackman's 10,000,000 word generator,

$ time tr '[[:blank:]]' '\n' < one.line | pr -s' ' -W 1000000 -JaT -1000 > many.line

real    0m2.113s
user    0m2.086s
sys 0m0.411s

where the counts are confirmed as follows

$ wc one.line multi.line 
        0  10000000  56608795 one.line
    10000  10000000  56608795 many.line
    10000  20000000 113217590 total

[Glenn's perl solution is still a little faster, ~1.8s on this machine].