Take the 2-minute tour ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.

Updated : Initial issue : Having a while read loop printing every line that is read Answer : Put a done <<< "$var" Subsequent issue : I may need some explanations about some SHELL code : I have this :

temp_ip=$($mysql --skip-column-names -h $db_address -u $db_user -p$db_passwd $db_name -e "select ip_routeur,code_site from $db_vtiger_table where $db_vtiger_table.ip_routeur NOT IN (select ip from $db_erreur_table);")

That gets results looking like this :

<ip1>   <site1>
<ip2>   <site2>
<ip3>   <site3>
<ip4>   <site4>

up to 5000 ip_address

I did a "while loop" :

while [ `find $proc_dir -name snmpproc* | wc -l` -ge "$max_proc_snmpget" ];do
    {
         echo "sleeping, fping in progress";
         sleep 1;
    }
done
temp_ip=$($mysql --skip-column-names -h $db_address -u $db_user -p$db_passwd $db_name -e "select ip_routeur,code_site from $db_vtiger_table where $db_vtiger_table.ip_routeur NOT IN (select ip from $db_erreur_table);")
while read ip codesite;do
{
    sendSNMPGET $ip $snmp_community $code_site &
}
done<<<"$temp_ip"

And the sendSNMPGET function is :

sendSNMPGET() {
touch $procdir/snmpproc.$$
hostname=`snmpget -v1 -c $2 $1 sysName.0`
if [ "$hostname" != "" ]
then
    echo "hi test"
fi
rm -f $procdir/snmpproc.$$

The $max_proc_snmpget is set to 30 At the execution, the read is ok, no more printing on screen, but child processes seems to be disoriented

hi
hi
hi
hi
hi
hi
hi
hi
hi
hi
hi
hi
./scan-snmp.sh: fork: Resource temporarily unavailable
./scan-snmp.sh: fork: Resource temporarily unavailable
./scan-snmp.sh: fork: Resource temporarily unavailable
./scan-snmp.sh: fork: Resource temporarily unavailable

Why can't it handle this ?

share|improve this question
    
It does it beacuse you are creating too many processes. Try looking at the value of $$ and seeing if it is changing... –  Mark Setchell May 22 at 10:44
add comment

4 Answers

up vote 0 down vote accepted

As you don't seem to have liked my answer with GNU Parallel, I'll show you a very simplistic way of doing it in parallel without needing to install that...

#!/bin/bash
MAX=8
j=0
while read ip code
do
   (sleep 5; echo $ip $code) &   # Replace this with your SNMPGET
   ((j++))
   if [ $j -eq $MAX ]; then
      echo -n Pausing with $MAX processes...
      j=0
      wait
   fi
done < file
wait

This starts up to 8 processes (you can change it) and then waits for them to complete before starting another 8. You have already been shown how to feed your mysql stuff into the loop by other respondents in the second to last line of the script...

The key to this is the wait which will wait for all started processes to complete.

share|improve this answer
    
Well, sorry i searched for Parallel, but not able to install it on Redhat. Plus this will cause some troubles when moving on the server (as i will not be present...) I tried your solution, it is working, i just had to change the done < file by done<<<"$var" (because of the printing) –  Gui O May 22 at 12:09
add comment

If temp_ip contains the name of a file that you want to read, then use:

done<"$temp_ip"

In your case, it appears that temp_ip is not a file name but contains the actual data that you want. In that case, use:

done<<<"$temp_ip"

Take care that the variable is placed inside double-quotes. That protects the data against the shell's word splitting which would result in the replacement of new line characters with spaces.

More details

In bash, an expression like <"$temp_ip" is called redirection. In this case in means that the while loop will get its standard input from the file called $temp_ip.

The expression <<<"$temp_ip" is called a here string. In this case, it means that the while loop will get its standard input from the data in the variable $temp_ip.

More information on both redirection and here strings in man bash.

share|improve this answer
    
If you can explain how <<< and what is the difference in < and <<< ! –  Jord May 22 at 8:00
    
echo Hello > a, then: cat < a will print Hello while cat <<< a will print a. –  Alfe May 22 at 8:06
    
Thanks, working ! But now having another issue... updated thread –  Gui O May 22 at 8:59
    
@Gui if your original problem is solved, please don't change your question beyond recognition. Leave this question so that the current answers are relevant and make a new question if you need help with something else. –  Tom Fenech May 22 at 9:01
    
Well, it's a subsequent issue with this answer, i guess it can be solved in the same thread... No ? –  Gui O May 22 at 9:07
show 1 more comment

Or you can parse the output of your initial command directly:

$mysql --skip-column-names -h $db_address -u $db_user -p$db_passwd $db_name -e "select ip_routeur,code_site from $db_vtiger_table where $db_vtiger_table.ip_routeur NOT IN (select ip from $db_erreur_table) | \
while read ip codesite
do
...
done

If you want to improve the performance and run some of the 5,000 SNMPGETs in parallel, I would recommend using GNU Parallel (here) like this:

$mysql --skip-column-names -h $db_address -u $db_user -p$db_passwd $db_name -e "select ip_routeur,code_site from $db_vtiger_table where $db_vtiger_table.ip_routeur NOT IN (select ip from $db_erreur_table) | parallel -k -j 20 -N 2 sendSNMPGET {1} $snmp_community {2}

The -k will keep the parallel output in order. The -j 20 will run up to 20 SNMPGETs in parallel at a time. The -N 2 means take 2 parameters from the mysql output per job (i.e. ip and codesite). {1} and {2} are your ip and codesite parameters.

http://www.gnu.org/software/parallel/

share|improve this answer
add comment

I propose to not store the result value but use it directly:

while read ip codesite
do
    sendSNMPGET "$ip" "$snmp_community" "$code_site" &
done < <(
  "$mysql" --skip-column-names -h "$db_address" -u "$db_user" -p"$db_passwd" "$db_name" \
         -e "select ip_routeur,code_site from $db_vtiger_table where $db_vtiger_table.ip_routeur NOT IN (select ip from $db_erreur_table);")

This way you start the mysql command in a subshell and use its output as input to the while loop (similar to piping which here also is an option).

But I see some problems with that code: If you really start each sendSNMPGET command in the background, you very quickly will put a massive load on your computer. For each line you read another active background process is started. This can slow down your machine to the point where it is rendered useless.

I propose to not run more than 20 background processes at a time.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.