current community

your communities

more stack exchange communities

Stack Exchange
tour help
Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.
up vote 1 down vote favorite

The following file runs, but not doing anything, yet it does not error.... 

while read dates; do ./avg_hrly_all_final.sh ${dates}; done < ./dates_all.csv

I have a list of dates in "dates_all.csv" that have the following form:

2005 01
2005 02
2005 03

And the script I am calling "avg_hrly_all_final.sh" works by passing it 2 positional parameters, example:

./avg_hrly_all_final.sh 2005 01

FOLLOW-UP

xargs -n2 ./the_script.sh <./dates_to_pass.csv

OR

while read dates; do ./the_script.sh ${dates}; done <./dates_to_pass.csv

work, just make sure the dates of the file being passed is of the same "End of Line" variety as the machine you are running the command on expects ;)

share|improve this question
    
As written above, your script should run ./avg_hrly_all_final.sh with two parameters. "no dice" So what actually happened when you ran the script? What made you think that the script's run was not successful? What error message did you see? –  John1024 Mar 1 at 8:20
    
The script that is supposed to be executed produces a set of files that did not populate. "No dice" is a colloquial term for "it didn't work". No error message. –  geokrowding Mar 1 at 9:37
1  
If using zsh, you need $=dates so $dates is split into 2 arguments. Other Bourne-like shells should do that implicitly unless you modified $IFS –  Stéphane Chazelas Mar 1 at 10:49

2 Answers 2

up vote 3 down vote accepted

This is a likely job for xargs:

printf %s\\n '#!/bin/sh' 'printf "<%s>\n" "$$" "$@"' >avg_hourly.sh
chmod +x ./avg_hourly.sh
xargs -n2 ./avg_hourly.sh <<\IN
2005 01
2005 02
2005 03
IN

xargs will split on the spaces by default and invoke the specified command once per -n2 occurring arguments. I just wrote a little dummy avg_hourly.sh script there which prints its arguments one per line as delimited at either end by < and > following its PID in the same format. The above prints:

<1115>
<2005>
<01>
<1116>
<2005>
<02>
<1117>
<2005>
<03>

...just to demonstrate. You should use <./dates_all.csv rather than my <<\IN here-document as input, though, probably.

share|improve this answer
    
Worked, thank you :) –  geokrowding Mar 1 at 10:09
up vote 0 down vote

If you have GNU Parallel you can do this:

parallel --colsep ' ' ./avg_hrly_all_final.sh {1} {2} :::: ./dates_all.csv

If the first line of the csv file is the header you can do something like:

parallel --header : --colsep ' ' ./avg_hrly_all_final.sh {Year} {Month} :::: ./dates_all.csv

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling

Installation

If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.