current community

your communities

more stack exchange communities

Stack Exchange
tour help
careers 2.0
Take the 2-minute tour ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
up vote 0 down vote favorite

i have a while loop reading lines from a $hosts

while read line
do
    ip=$line
    check
done < $hosts

my question is can I use some way to speed this up or run the check on 10 hosts at a time and each check is on a different IP and finish when all IP in $host have been checked? Thanks

share|improve this question

2 Answers 2

up vote 0 down vote accepted

You can send tasks to the background by & If you intend to wait for all of them to finish you can use the wait command:

process_to_background &
echo Processing ...
wait
echo Done

You can get the pid of the given task started in the background if you want to wait for one (or few) specific tasks.

important_process_to_background &
important_pid=$!
while i in {1..10}; do
    less_important_process_to_background $i &
done

wait $important_pid
echo Important task finished

wait
echo All tasks finished

On note though: the background processes can mess up the output as they will run asynchronously. You might want to use a named pipe to collect the output from them.

edit

As asked in the comments there might be a need for limiting the background processes forked. In this case you can keep track of how many background processes you've started and communicate with them through a named pipe.

mkfifo tmp # creating named pipe

counter=0
while read ip
do
  if [ $counter -lt 10 ]; then # we are under the limit
    { check $ip; echo 'done' > tmp; } &
    let $[counter++];
  else
    read x < tmp # waiting for a process to finish
    { check $ip; echo 'done' > tmp; } &
  fi
done
cat /tmp > /dev/null # let all the background processes end

rm tmp # remove fifo
share|improve this answer
    
Will this cycle through every host in $hosts? Even if there is more than 10? I don't want to start any more than 10 processes it could start taking up too much resources. –  radman Apr 13 at 22:41
    
This is just an example how you can use background processes and how you can wait for them. If you want to specify the maximum number of threads that sounds like a thread pool. For that I add an example, however it might not be the best implementation –  fejese Apr 13 at 23:11
up vote 0 down vote

You can start multiple processes, each calling the function check and wait for them to finish.

while read line 
do 
  ip=$line
  check &
done < $hosts
wait # wait for all child processes to finish

Whether this increases the speed depends on available processors and the function check's implementation. You have to ensure there's no data dependency in check between iterations.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.