Running commands in parallel, with or without GNU parallel. For virtualization on Macs, see /parallels. For parallel ports, see /parallel-port.
3
votes
3answers
67 views
Bash, how to let some background processes run but wait for others?
I have (yet) another wait, &, && control flow question..
Say I have a script something like this where I want to do as much work at the same time as possible:
# may take some hours
...
6
votes
4answers
362 views
How to run parallel processes and combine outputs when both finished
I have a bash shell script in which I pipe some data through about 5 or 6 different programs then the final results into a tab delimited file.
I then do the same again for a separate similar dataset ...
8
votes
4answers
99 views
Virtual write-only file system for storing files in archive
I have an embarrassingly parallel process that creates a huge amount of nearly (but not completely) identical files. Is there a way to archive the files "on the fly", so that the data does not consume ...
2
votes
2answers
54 views
Background execution in parallel
I have a program ./pgm taking some arguments (say -a file1 -b val), which requires 2 seconds to execute. I would like to use all the processors on my machine to run this program on all the input files ...
3
votes
3answers
111 views
How to parallelize the for loop while scp the files?
I am running my below shell script from machineA which is copying the files machineB and machineC into machineA. If the files are not there in machineB, then it should be there in machineC.
The below ...
1
vote
1answer
37 views
Gnu Parallel: more than one per CPU? [closed]
If I only have two cores, does parallel only run two processes at a time? I guess I'm still confused about processes, threads, and cores....
1
vote
1answer
26 views
Is it safe to resize a partition while writing to it?
I'm working on a script for automatically setting up Amazon Linux servers. I create them with 100gb virtual disks, but the main partition is always 8gb. No problem, I call sudo resize2fs /dev/sda1 at ...
0
votes
3answers
42 views
running an executable in parallel by dividing the input directory tree
Commands like find and grep would benefit greatly if they were to run in parallel. Is there a utility or a program that takes a command as an argument and makes it run in parallel by running multiple ...
2
votes
1answer
126 views
Can I force Linux to boot its initializations parallel? [closed]
I was wondering if I could manage to initialize drivers, services etc. (all these jobs what Linux does during startup) in parallel instead of sequentially. I want to aggressively lower the boot time. ...
5
votes
2answers
85 views
How can I parallelise the upload of a directory by FTP?
I need to upload a directory with a rather complicated tree (lots of subdirectories, etc.) by FTP. I am unable to compress this directory, since I do not have any access to the destination apart from ...
-1
votes
1answer
48 views
GNU Parallel slows down
I'm learning GNU parallel usage and did the following:
$ for i in {1.txt,2.txt}; do time wc -l $i; done
100 1.txt
real 0m0.010s
user 0m0.000s
sys 0m0.010s
10000012 2.txt
real 0m0.069s
...
1
vote
1answer
37 views
Memory-efficient utility to return N first sorted values
I would like to implement a very popular MapReduce example using only existing programs operating in a UNIX way. The problem is to find N most frequent values in an enormous amount of data. The ...
2
votes
1answer
21 views
No output using parallel in tandem with ag or ack
I have a list of java reserved words, first letter capitalised.
$ tail -n5 ~/reservedjava.txt
Break
While
True
False
Null
I'm trying to look through all my java source code to find methods that ...
1
vote
4answers
235 views
Ctrl-C with two simultaneous commands in bash
I want to run two commands simultaneously in bash on a Linux machine. Therefore in my ./execute.sh bash script I put:
command 1 & command 2
echo "done"
However when I want to stop the bash ...
0
votes
1answer
69 views
ERROR: cube viewer is not available!
I have installed and configured the following:
cube-4.2.1
opari2-1.1.1
papi-5.2.0
scalasca-2.0
scorep-1.2.1
I am trying to run scalasca -examine, but I get the following error:
ERROR: cube viewer ...
2
votes
4answers
58 views
Parallel pausing and resuming?
I have a pretty strange thing I'm seeing as I run 8 GPG encryption jobs with GNU parallel:
The command I've run is this:
find . -type f -not -iname "*.gpg" | sort |parallel --gnu -j 8 --workdir ...
7
votes
1answer
148 views
GNU parallel vs & (I mean background) vs xargs -P
I'm confused about the difference or advantage (if any) of running a set of tasks in a .sh script using GNU parallel
E.g. Ole Tange's answer:
parallel ./pngout -s0 {} R{} ::: *.png
rather than say ...
2
votes
3answers
154 views
Parallelize a Bash FOR Loop
I have been trying to parallelize the following script, specifically each of the three FOR loop instances, using GNU Parallel but haven't been able to. The 4 commands contained within the FOR loop run ...
2
votes
2answers
137 views
scp does not display output when used with gnu parallel
I have a short script that uses scp to copy files to a number of remote hosts (yes, I know about rdist and rsync; they both fail to work for a few of the hosts - that's not the point here; I'm only ...
0
votes
0answers
33 views
log4cxx/logger.h or libcql/cql.hpp fatal error while doing make -j4
I am running make -j4 and I am getting below exception...
userName@phx5qa01c-4e23:~/build$ make -j4
[ 1%] [ 2%] Building CXX object nda/common/loadtest/CMakeFiles/loadtester.dir/LoadTester.cc.o
...
1
vote
1answer
115 views
Is there a simple way of limiting the number of child processes that execute in parallel?
I have a bash script that kicks off multiple rsync processes in parallel. I'd like to limit the number of simultaneous child processes but not serialize the whole script.
Here's a simplified version ...
1
vote
1answer
59 views
Strategies for building distributed computing system with hybrid Mac and Linux systems
I want to build a distributed computing system to run Matlab, C and other programming languages for scientific computing. Now I've several old Mac machines with Lion Mac OS installed acted as web ...
3
votes
1answer
145 views
How can I issue parallel commands to remote nodes with different arguments?
I need to execute an application in parallel on multiple Ubuntu servers while supplying different arguments for different servers. I tried to google it, but could not get to the possible solution. I ...
2
votes
8answers
539 views
How do I run the same linux command in more than one tab/shell simultaneously?
Is there any tool/command in Linux that I can use to run a command in more than one tab simultaneously? I want to run the same command: ./myprog argument1 argument2 simultaneously in more than one ...
0
votes
1answer
539 views
Rsync from multiple folders to one
Is it safe to have multiple folders Rsync simultaneously into one?
Each of the source folders contains a subset of the destination folder, and everybody just syncs into a master destination folder. I ...
3
votes
1answer
96 views
Variable declaration in parallel sh -c …
I've been trying to process the output of find with parallel, which in turn invoked a shell (some textual substitutions were needed). I observed some strange behaviour, which I cannot really explain ...
3
votes
2answers
129 views
Running up to X commands in parallel
I'm running something like this:
find . -maxdepth 1 -type f -note -iname "*.gpg" | sort | while read file ; do
echo "Encrypting $file..."
gpg --trust-model always --recipient "[email protected]" ...
7
votes
2answers
892 views
Executing piped commands in parallel
Consider the following scenario. I have two programs A and B. Program A outputs to stdout lines of strings while program B process lines from stdin. The way to use these two programs is of course
...
1
vote
4answers
104 views
How can i run a piece of code in background?
Can I run a piece of code in background instead of using another script.
[sesiv@itseelm-lx4151 ~]$ cat ./testback2
#!/bin/bash
start_time=$(date +%s)
for i in {1..5}
do
./testscript &
done
wait
...
2
votes
1answer
79 views
replacement inside parallel command string
I want to download multiple files and save them to the md5 sum of its name:
cat list | parallel -j4 "md5=$(wget -O - {} | tee tmpfile_{#} | md5sum | cut -d ' ' -f 1); mv tmpfile_{#} $md5"
but the ...
4
votes
4answers
413 views
Using GNU Parallel With Split
I'm loading a pretty gigantic file to a postgresql database. To do this I first use split in the file to get smaller files (30Gb each) and then I load each smaller file to the database using GNU ...
1
vote
0answers
237 views
MPI mpdboot error
I have a problem with mpi in a 3-box cluster with machines named head, node1 and node2 respectively. I install a newer version on node2 and the problems arises. When I include only head and node2 in ...
5
votes
1answer
384 views
Running parallel bash jobs on a HPC cluster using GNU parallel
On an HPC cluster I am trying to run multiple bash scripts (permute2.sh) from 1 bash script using GNU parallel, however it doesn't complete every job. It randomly completes one job, while it is stuck ...
2
votes
1answer
178 views
Is there a way to run process parallelly in the loop of a bash script [duplicate]
Possible Duplicate:
Parallelizing a for loop
The original code might look like this:
for i in *; do something.py $i; done
I was wondering whether I can run these jobs parallelly in the ...
3
votes
3answers
436 views
Copy multiple files to one dir with parallel
I'm using the following script to copy multiple files into one folder:
{ echo $BASE1; echo $BASE2; echo $BASE3; } | parallel cp -a {} $DEST
Is there any way to use only one echo $BASE with brace ...
7
votes
4answers
479 views
Creating a single output stream out of three other streams produced in parallel
I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format.
This Python script is slow and CPU-bound (to a ...
11
votes
8answers
703 views
Spreading stdin to parallel processes
I have a task that processes a list of files on stdin. The start-up time of the program is substantial, and the amount of time each file takes varies widely. I want to spawn a substantial number of ...
2
votes
3answers
237 views
control number of started programs in bash
As part of my research project I'm processing huge amount of data splitted up into many files.
All files in folder foo have to be processed by the script myScript involving all elements of folder ...
4
votes
3answers
1k views
shell script to read from multiple files in parallel
I need to write a script that runs parallel and looks for a string in multiple files. I tried a lot of options but they slow down the speed of my processor.
3
votes
2answers
321 views
Running jobs in parallel on Ubuntu - I/O contention differences between Perl and Java
Apologies if this is off topic - it concerns the relative efficiencies of running I/O-heavy Perl/Java scripts in parallel on a Ubuntu system.
I have written two simple versions of a file copy script ...
4
votes
2answers
300 views
`mpirun -np N`: what if `N` is larger than my physical cores?
Say I have a 4-core workstation, what would Linux (Ubuntu) do if I execute
mpirun -np 9 XXX
Will 9 run immediately together, or they will run 4 after 4?
I suppose that using 9 is not good, because ...
4
votes
4answers
2k views
Remove numbers from filenames
I've a problem modifying the files' names in my Music/ directory.
I have a list of names like these:
$ ls
01 American Idiot.mp3
01 Articolo 31 - Domani Smetto.mp3
01 Bohemian rapsody.mp3
01 Eye of ...
6
votes
4answers
3k views
Parallel execution of a program on multiple files
I have a small script that loops through all files of a folder and executes a (usually long lasting) command. Basically it's
for file in ./folder/*;
do
./bin/myProgram $file > ./done/$file
...
4
votes
6answers
3k views
Is there parallel wget? Something like fping but only for downloading?
I've found only puf (Parallel URL fetcher) but could to get it work with reading urls from file and something like
puf < urls.txt
does not work either.
The operating system installed on the ...
9
votes
2answers
1k views
Why does (GNU?) parallel fail silently, and how do I fix it?
In a larger script to post-process some simulation data I had the following line:
parallel bnzip2 -- *.bz2
Which, if I understand parallel correctly (and I may not), should run n-core threads of ...
15
votes
4answers
3k views
Four tasks in parallel… how do I do that?
I have a bunch of PNG images on a directory. I have an application called pngout that I run to compress these images. This application is called by a script I did. The problem is that this script does ...
3
votes
3answers
2k views
process files in a directory as they appear [duplicate]
Possible Duplicate:
How to run a command when a directory's contents are updated?
I'm trying to write a simple etl process that would look for files in a directory each minute, and if ...
4
votes
1answer
172 views
Multi-machine tool in the spirit of moreutils' `parallel`?
parallel from moreutils is a great tool for, among other things, distributing m independent tasks evenly over n CPUs. Does anybody know of a tool that accomplishes the same thing for multiple ...
6
votes
4answers
2k views
using parallel to process unique input files to unique output files
I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to ...
2
votes
3answers
296 views
Are there any disadvantages against using qsub to run tasks all the time?
When I'm running a task on a computer network? I've just started to realize that if I qsub any task, then the task won't hog up my terminal, and I can do other things on the same terminal (which is ...
