Tagged Questions

Running commands in parallel, with or without GNU parallel. For virtualization on Macs, see /parallels. For parallel ports, see /parallel-port.

learn more… | top users | synonyms

1
vote
8answers
149 views

How do I run the same linux command in more than one tab/shell simultaneously?

Is there any tool/command in Linux that I can use to run a command in more than one tab simultaneously? I want to run the same command: ./myprog argument1 argument2 simultaneously in more than one ...
0
votes
1answer
46 views

Rsync from multiple folders to one

Is it safe to have multiple folders Rsync simultaneously into one? Each of the source folders contains a subset of the destination folder, and everybody just syncs into a master destination folder. I ...
3
votes
1answer
48 views

Variable declaration in parallel sh -c …

I've been trying to process the output of find with parallel, which in turn invoked a shell (some textual substitutions were needed). I observed some strange behaviour, which I cannot really explain ...
3
votes
2answers
59 views

Running up to X commands in parallel

I'm running something like this: find . -maxdepth 1 -type f -note -iname "*.gpg" | sort | while read file ; do echo "Encrypting $file..." gpg --trust-model always --recipient "[email protected]" ...
7
votes
2answers
302 views

Executing piped commands in parallel

Consider the following scenario. I have two programs A and B. Program A outputs to stdout lines of strings while program B process lines from stdin. The way to use these two programs is of course ...
1
vote
4answers
72 views

How can i run a piece of code in background?

Can I run a piece of code in background instead of using another script. [sesiv@itseelm-lx4151 ~]$ cat ./testback2 #!/bin/bash start_time=$(date +%s) for i in {1..5} do ./testscript & done wait ...
2
votes
1answer
55 views

replacement inside parallel command string

I want to download multiple files and save them to the md5 sum of its name: cat list | parallel -j4 "md5=$(wget -O - {} | tee tmpfile_{#} | md5sum | cut -d ' ' -f 1); mv tmpfile_{#} $md5" but the ...
4
votes
4answers
258 views

Using GNU Parallel With Split

I'm loading a pretty gigantic file to a postgresql database. To do this I first use split in the file to get smaller files (30Gb each) and then I load each smaller file to the database using GNU ...
1
vote
0answers
157 views

MPI mpdboot error

I have a problem with mpi in a 3-box cluster with machines named head, node1 and node2 respectively. I install a newer version on node2 and the problems arises. When I include only head and node2 in ...
4
votes
1answer
278 views

Running parallel bash jobs on a HPC cluster using GNU parallel

On an HPC cluster I am trying to run multiple bash scripts (permute2.sh) from 1 bash script using GNU parallel, however it doesn't complete every job. It randomly completes one job, while it is stuck ...
1
vote
1answer
127 views

Is there a way to run process parallelly in the loop of a bash script [duplicate]

Possible Duplicate: Parallelizing a for loop The original code might look like this: for i in *; do something.py $i; done I was wondering whether I can run these jobs parallelly in the ...
3
votes
3answers
321 views

Copy multiple files to one dir with parallel

I'm using the following script to copy multiple files into one folder: { echo $BASE1; echo $BASE2; echo $BASE3; } | parallel cp -a {} $DEST Is there any way to use only one echo $BASE with brace ...
7
votes
4answers
353 views

Creating a single output stream out of three other streams produced in parallel

I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format. This Python script is slow and CPU-bound (to a ...
6
votes
5answers
462 views

Spreading stdin to parallel processes

I have a task that processes a list of files on stdin. The start-up time of the program is substantial, and the amount of time each file takes varies widely. I want to spawn a substantial number of ...
2
votes
3answers
181 views

control number of started programs in bash

As part of my research project I'm processing huge amount of data splitted up into many files. All files in folder foo have to be processed by the script myScript involving all elements of folder ...
4
votes
3answers
854 views

shell script to read from multiple files in parallel

I need to write a script that runs parallel and looks for a string in multiple files. I tried a lot of options but they slow down the speed of my processor.
3
votes
2answers
254 views

Running jobs in parallel on Ubuntu - I/O contention differences between Perl and Java

Apologies if this is off topic - it concerns the relative efficiencies of running I/O-heavy Perl/Java scripts in parallel on a Ubuntu system. I have written two simple versions of a file copy script ...
4
votes
2answers
234 views

`mpirun -np N`: what if `N` is larger than my physical cores?

Say I have a 4-core workstation, what would Linux (Ubuntu) do if I execute mpirun -np 9 XXX Will 9 run immediately together, or they will run 4 after 4? I suppose that using 9 is not good, because ...
3
votes
4answers
1k views

Remove numbers from filenames

I've a problem modifying the files' names in my Music/ directory. I have a list of names like these: $ ls 01 American Idiot.mp3 01 Articolo 31 - Domani Smetto.mp3 01 Bohemian rapsody.mp3 01 Eye of ...
6
votes
4answers
2k views

Parallel execution of a program on multiple files

I have a small script that loops through all files of a folder and executes a (usually long lasting) command. Basically it's for file in ./folder/*; do ./bin/myProgram $file > ./done/$file ...
2
votes
6answers
2k views

Is there parallel wget? Something like fping but only for downloading?

I've found only puf (Parallel URL fetcher) but could to get it work with reading urls from file and something like puf < urls.txt does not work either. The operating system installed on the ...
8
votes
2answers
747 views

Why does (GNU?) parallel fail silently, and how do I fix it?

In a larger script to post-process some simulation data I had the following line: parallel bnzip2 -- *.bz2 Which, if I understand parallel correctly (and I may not), should run n-core threads of ...
13
votes
4answers
2k views

Four tasks in parallel… how do I do that?

I have a bunch of PNG images on a directory. I have an application called pngout that I run to compress these images. This application is called by a script I did. The problem is that this script does ...
3
votes
3answers
2k views

process files in a directory as they appear [duplicate]

Possible Duplicate: How to run a command when a directory's contents are updated? I'm trying to write a simple etl process that would look for files in a directory each minute, and if ...
3
votes
1answer
146 views

Multi-machine tool in the spirit of moreutils' `parallel`?

parallel from moreutils is a great tool for, among other things, distributing m independent tasks evenly over n CPUs. Does anybody know of a tool that accomplishes the same thing for multiple ...
6
votes
4answers
1k views

using parallel to process unique input files to unique output files

I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to ...
2
votes
3answers
213 views

Are there any disadvantages against using qsub to run tasks all the time?

When I'm running a task on a computer network? I've just started to realize that if I qsub any task, then the task won't hog up my terminal, and I can do other things on the same terminal (which is ...
3
votes
3answers
321 views

How to create a bounded queue for shell tasks?

I have 1000 gzipped files which I want to sort. Doing this sequentially, the procedure looks pretty straightforward: find . -name *.gz -exec zcat {} | sort > {}.txt \; Not sure that the code ...
2
votes
4answers
875 views

Parallelizing a for loop with very large number of iterations

I want to parallelize a for loop where the number of iterations in the loop can be very large such as 10^6. So ,it will be better if I can create threads rather than process. How to do it? The code is ...
7
votes
2answers
2k views

Parallelizing a for loop

I want to parallelize the for loops of the following code. How to do this? #!/bin/bash N=$1 n=$2 for (( i=1; i<=$N; i++ )); do min=100000000000000 //set min to some garbage value for (( ...
2
votes
1answer
699 views

How to run Abinit via MPI?

I'm trying to run Abinit in parallel, since in sequential mode it takes too much time. I have followed all steps in the abinit tutorials to run it in parallel, and they say to use OpenMPI, but I ...
8
votes
2answers
9k views

Limit process to one cpu core

How to limit process to one cpu core ? Something similar to ulimit or cpulimit would be nice. (Just to ensure: I do NOT want to limit percentage usage or time of execution. I want to force app (with ...
5
votes
3answers
443 views

How to speed up my build

I am doing a build on a Linux machine with ubuntu 10.04 on it. I want to know how can I really speed up my build. I have 4 CPUs and lots of RAM. I already reniced the process group to -20. Is there ...
4
votes
1answer
3k views

Should I disable hyperthreading when concerned about performance of single-threaded applications?

I use a i5-2410M processor, which is setup to do hyperthreading by default on my laptop. Considering that this is a 2-core processor, this means it can do 4 threads at a time. This also means that ...
6
votes
2answers
985 views

How to stop xargs from badly merging output from multiple processes?

I'm using xargs with the option --max-args=0 (alternatively -P 0). However, the output of the processes is merged into the stdout stream without regard for proper line separation. So I'll often end ...
5
votes
1answer
386 views

Poor Man's GNU Parallel implemented in ksh?

I'd like to use the feature of GNU parallel where it can execute the command and the list it's fed in parallel and spit it out after it's all done, however, I don't want to install GNU parallel across ...
4
votes
2answers
493 views

Forcing GNU make to run commands in order

With the following Makefile, GNU make runs the two commands in parallel. Since the first one takes time to finish, rm *.log is run before the log file is created, and fails. dummy.pdf: dummy.tex ...
4
votes
3answers
921 views

Using parallel on Ubuntu

I am having an issue trying to use parallel command on Ubuntu 10.04. I looked up the parallel documentation and few of the commands seem to run. In all cases I just get the command prompt back without ...