current community

your communities

more stack exchange communities

company blog
Stack Exchange Inbox Reputation and Badges
tour help
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top
up vote 8 down vote favorite
3

I have written a bash script which is in following format:

#!/bin/bash
start=$(date +%s)
inFile="input.txt"
outFile="output.csv"

rm -f $inFile $outFile

while read line
do

    -- Block of Commands

done < "$inFile"

end=$(date +%s)

runtime=$((end-start))

echo "Program has finished execution in $runtime seconds."

The while loop will read from $inFile, perform some activity on the line and dump the result in $outFile.

As the $inFile is 3500+ lines long, the script would take 6-7 hours for executing completely. In order to minimize this time, I am planning to use multi-threading or forking in this script. If I create 8 child processes, 8 lines from the $inFile would be processed simultaneously.

How can this be done?

share|improve this question
    
Be careful: different scripts will need to write to different outfiles. Also your script as written deletes the input file as the first action! – pjc50 Aug 1 '14 at 10:59
up vote 9 down vote accepted

GNU parallel is made for just this sort of thing. You can run your script many times at once, with different data from your input piped in for each one:

cat input.txt | parallel --pipe your-script.sh

By default it will spawn processes according to the number of processors on your system, but you can customise that with -j N.

A particularly neat trick is the shebang-wrapping feature. If you change the first line of your Bash script to:

#!/usr/bin/parallel --shebang-wrap --pipe /bin/bash

and feed it data on standard input then it will all happen automatically. This is less useful when you have cleanup code that has to run at the end, which you may do.

There are a couple of things to note. One is that it will chop up your input into sequential chunks and use those one at a time - it doesn't interleave lines. The other is that it those chunks are split by size, without regard for how many records there are. You can use --block N to set a different block size in bytes. In your case, no more than an eighth of the file size should be about right. Your file sounds like it might be small enough to end up all in a single block otherwise, which would defeat the purpose.

There are a lot of options for particular different use cases, but the tutorial covers things pretty well. Options you might also be interested in include --round-robin and --group.

share|improve this answer
1  
Did you test that shebang line? Shebangs with multiple arguments are unportable. On Linux, #!a b c will result in ["b c"], while on some other systems, it will result in ["b", "c"]. – nyuszika7h Aug 1 '14 at 14:33
1  
It reparses its own arguments when used in this way (otherwise the option wouldn't be much use). – Michael Homer Aug 1 '14 at 18:58
    
@MichaelHomer I need to use GNU parallel for scraping of HTML pages. Could you please go through this thread unix.stackexchange.com/questions/277609/… – Swatesh Pakhare Apr 19 at 23:26

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.