AMBIGUOUS TAG — DO NOT USE. For the Mac virtualization software, use [parallels]. For the `parallel` utilities, use [gnu-parallel] or [moreutils-parallel]. For parallel computing in general, use [parallelism]. For parallel ports once common on PCs, use [parallel-port].
1
vote
0answers
24 views
It is possible to run these in GNU parallel [closed]
I would like to run these:
screen watch -n 0 php /usr/share/nginx/html/2siti.php &
screen watch -n 0 php /usr/share/nginx/html/2siti.php &
screen watch -n 0 php ...
2
votes
1answer
31 views
Find | parallel executing script with path from find + other arguments
Due to my lack of Bash knowledge I've tried for a few hours now to get something like this to work:
find Directories -mindepth 4 -type d -print0 | parallel -0 -j0 ./MyScript -d {Found Directory} {1} ...
1
vote
3answers
66 views
tarring in parallel
Ab Oceanographer friend at work needs to back up many months worth of data. She is overwhelemed so I volunteered to do it. There are hundreds of directories to be backed up and we want to tar/bzip ...
0
votes
0answers
74 views
Parallel vs Distributed vs Traditional File system
I am trying to understand the difference between these three file system at a very basic level.
Distributed FS: HDFS
Parallel FS : Lustre
Tradational FS : ext4/ext3/ NTFS/FAT etc.
I want to ...
7
votes
3answers
561 views
How to start multi-threaded grep in terminal?
I have a folder which has 250+ files of 2 GB each. I need to search for a string/pattern in those files and output the result in an output file. I know I can run the following command, but it is too ...
1
vote
2answers
44 views
Run multiple scripts and continue until every script is done
my question is, can I run multiple scripts at once. I have access to a cluster with 32 cpus but at the same time I have 900 scripts. Now what I would like to do is, put every script in a queue, and ...
0
votes
1answer
134 views
Few questions about system calls and kernel modules (kernel services in parallel)
I have few questions about system calls and kernel modules.
Let's assume we have two applications (A and B) and each of them is running on a different core. (A-CPU 0, B-CPU 1)
When two applications ...
3
votes
3answers
132 views
Run fgrep multicore mac os x
I'm running this command on my Mac OS X on 8 cores:
for i in $(cat file1);do grep "$i" file2; done > output.txt
My file2 is 9 million lines so it takes a lot of time. Is it possible to use all ...
0
votes
0answers
17 views
PBS limiting the number of process on a node
I am trying to run a program on cluster. The program can distribute each of the calculation with an corresponding parameter, say k1, k2,...k4, to run on one core. I want to use 4 nodes with 1 ...
3
votes
1answer
190 views
GNU Parallel Limit Memory Usage
Is it possible to limit the memory usage of all processes started by GNU parallel? I realize there are ways to limit the number of jobs, but in cases where it isn't easy to predict the memory usage ...
0
votes
1answer
1k views
Is it possible to run two commands at the same time in a shell script?
I know that on the command line I can use & to run a command in the background. But I'm wondering if I can do it in a script.
I have a script like this:
date_stamp=$(date +"%Y-%m-%d" ...
1
vote
1answer
362 views
How to launch multiple instances of a program in parallel using parallel?
I would like to execute a command:
curl example.com
multiple times in parallel using GNU parallel. I tried:
parallel -j 3 curl example.com
but it doesn't work.
1
vote
2answers
82 views
How can I count the time that needs systemd to boot a default target and then graph it?
When I wanted to know how long systemd actually needed to boot the default target, how would I do that ? An then, is it possible to create a graph to show which unit takes how much time to initialize ...
0
votes
1answer
263 views
Running shell script on cluster in parralel
I thought dsh(Distributed Shel) will be a good option but when i ran my shell script across my nodes, I didnt get expected output,
dsh -aM -c bash /home/cloudera/bash_script.sh
[email protected]: ...
1
vote
1answer
47 views
Wine and parallel processing
I'm quite new to linux, right now, running LMDE on my machine.
I was wondering if wine can support the parallel processing needs of some software.
Anyone knows any performance benchmark of wine ?
...
4
votes
1answer
16k views
top command on multi core processor
I am using freescale IMX6 quad processor. I want to know if the top command lists the CPU usage of all 4 cores or of a single core. I am seeing an application's CPU usage being the same with 4 cores ...
3
votes
4answers
13k views
Number of processors in /proc/cpuinfo
While i was learning about cpu load i came to know that it depends on numbers of core. if i have 2 cores then load 2 will give 100% cpu utilization.
So i tried to find out cores.( i already know that ...
3
votes
3answers
321 views
Is the output mixed like this because of xargs and how can I fix it?
I'm learning a bit of Bash by using xargs to list whois records for a bunch of IP addresses. The command used is:
echo "$1" | tr "\n" "\0" | xargs -0 -n 1 -t -P 3 -I % sh -c 'echo "\n 44rBegin whois ...
2
votes
2answers
323 views
How to read commands from file?
I have som php scripts which I run sequential order like:
php index.php import file1 --offline &&
php index.php import file2 --deleteUnused &&
php index.php import file3
Now I have ...
6
votes
3answers
630 views
Multiple tar processes writing to the same archive file at once
I am running many tasks on a Linux cluster. Each task creates many output files. When all tasks are finished, I run something like tar cf foo.tar output_files/to create a tar archive. This is a very ...
2
votes
2answers
306 views
How does a parallelized Python program look with top command?
I work on a shared cluster. I've seen people run parallelized c code on this cluster which, when I use top to see what processes are running, are shown to be using (for example) 400% of the CPU, ...
1
vote
2answers
8k views
MaxStartups and MaxSessions configurations parameter for ssh connections?
I am copying the files from machineB and machineC into machineA as I am running my below shell script on machineA.
If the files is not there in machineB then it should be there in machineC for sure ...
6
votes
3answers
305 views
Is it safe to pipe the stdout of A to B in parallel
I'm running A file | B --params > file.txt. Since I want to accelerate the processing speed, I used the parallel -j+0< a.txt to run 20 jobs concurrently. a.txt contains all the commands:
A ...
1
vote
1answer
177 views
How to copy in two folders simultaneously using GNU parallel by spawning multiple threads?
I am trying to copy files from machineB and machineC into machineA as I am running my below shell script on machineA.
If the files is not there in machineB then it should be there in machineC for ...
3
votes
3answers
3k views
parallel-ssh with Passphrase Protected SSH Key
I'd like to use my passphrase protected SSH key when performing tasks with parallel-ssh (and related tools). However, I can't get this to work.
All the documentation, regarding parallel-ssh, show ...
2
votes
2answers
546 views
Variable value is not recognized after using gnu parallel?
I have a below shell script from which I am trying to copy 5 files in parallel. I am running my below shell script on machineA which tries to copy the file from machineB and machineC.
If the file is ...
2
votes
3answers
149 views
How to split the array in set of five files and download them in parallel?
I am trying to copy files from testMachineB and testMachineC into testMachineA as I am running my shell script on testMachineA.
If the file is not there in testMachineB, then it should be there in ...
2
votes
1answer
371 views
How to get exit code of a parallel process launched by xargs?
I'd made a script for launching parallel rsync process:
#! /bin/bash
LIST=$1
DEST_DIR=$2
RSYNC_OPTS=$3
#echo "rsyncing From=$SRC_DIR To=$DEST_DIR RSYNC_OPTS=$RSYNC_OPTS"
echo $LIST|xargs -n1 -d, ...
2
votes
2answers
502 views
Can GNU Parallel execute more parallel processes?
Can I for example execute:
parallel -j 200 < list0
Where "list" has:
nice -n -20 parallel -j 100 < list2
nice -n -20 parallel -j 100 < list1
Would this be feasible/possible?
1
vote
1answer
107 views
Starting GNU Parallel in background?
Is it possible to start GNU Parallel in the background, or somehow, send it to the background?
I want to do this so that I can use two commands that run parallel at the same time.
And why dont I ...
1
vote
1answer
135 views
Why does parallel slow down after a while?
I'm executing 60 scripts with GNU parallel(they all have wget commands in there)but I have noticed that after a few hours execution will slow down a bit. What could be causing this?
I'm executing ...
6
votes
4answers
8k views
Bash, how to let some background processes run but wait for others?
I have (yet) another wait, &, && control flow question..
Say I have a script something like this where I want to do as much work at the same time as possible:
# may take some hours
...
9
votes
4answers
5k views
How to run parallel processes and combine outputs when both finished
I have a bash shell script in which I pipe some data through about 5 or 6 different programs then the final results into a tab delimited file.
I then do the same again for a separate similar dataset ...
8
votes
4answers
251 views
Virtual write-only file system for storing files in archive
I have an embarrassingly parallel process that creates a huge amount of nearly (but not completely) identical files. Is there a way to archive the files "on the fly", so that the data does not consume ...
2
votes
2answers
96 views
Background execution in parallel
I have a program ./pgm taking some arguments (say -a file1 -b val), which requires 2 seconds to execute. I would like to use all the processors on my machine to run this program on all the input files ...
3
votes
3answers
1k views
How to parallelize the for loop while scp the files?
I am running my below shell script from machineA which is copying the files machineB and machineC into machineA. If the files are not there in machineB, then it should be there in machineC.
The below ...
1
vote
1answer
405 views
Gnu Parallel: more than one per CPU? [closed]
If I only have two cores, does parallel only run two processes at a time? I guess I'm still confused about processes, threads, and cores....
1
vote
1answer
47 views
Is it safe to resize a partition while writing to it?
I'm working on a script for automatically setting up Amazon Linux servers. I create them with 100gb virtual disks, but the main partition is always 8gb. No problem, I call sudo resize2fs /dev/sda1 at ...
0
votes
3answers
295 views
running an executable in parallel by dividing the input directory tree
Commands like find and grep would benefit greatly if they were to run in parallel. Is there a utility or a program that takes a command as an argument and makes it run in parallel by running multiple ...
2
votes
1answer
807 views
Can I force Linux to boot its initializations parallel? [closed]
I was wondering if I could manage to initialize drivers, services etc. (all these jobs what Linux does during startup) in parallel instead of sequentially. I want to aggressively lower the boot time. ...
6
votes
2answers
1k views
How can I parallelise the upload of a directory by FTP?
I need to upload a directory with a rather complicated tree (lots of subdirectories, etc.) by FTP. I am unable to compress this directory, since I do not have any access to the destination apart from ...
0
votes
1answer
181 views
Why does GNU Parallel slow down?
I'm learning GNU parallel and tried the following:
$ for i in {1.txt,2.txt}; do time wc -l $i; done
100 1.txt
real 0m0.010s
user 0m0.000s
sys 0m0.010s
10000012 2.txt
real 0m0.069s
user ...
1
vote
1answer
72 views
Memory-efficient utility to return N first sorted values
I would like to implement a very popular MapReduce example using only existing programs operating in a UNIX way. The problem is to find N most frequent values in an enormous amount of data. The ...
3
votes
1answer
271 views
No output using parallel in tandem with ag or ack
I have a list of java reserved words, first letter capitalised.
$ tail -n5 ~/reservedjava.txt
Break
While
True
False
Null
I'm trying to look through all my java source code to find methods that ...
0
votes
1answer
149 views
ERROR: cube viewer is not available!
I have installed and configured the following:
cube-4.2.1
opari2-1.1.1
papi-5.2.0
scalasca-2.0
scorep-1.2.1
I am trying to run scalasca -examine, but I get the following error:
ERROR: cube viewer ...
2
votes
4answers
141 views
Parallel pausing and resuming?
I have a pretty strange thing I'm seeing as I run 8 GPG encryption jobs with GNU parallel:
The command I've run is this:
find . -type f -not -iname "*.gpg" | sort |parallel --gnu -j 8 --workdir ...
15
votes
1answer
3k views
GNU parallel vs & (I mean background) vs xargs -P
I'm confused about the difference or advantage (if any) of running a set of tasks in a .sh script using GNU parallel
E.g. Ole Tange's answer:
parallel ./pngout -s0 {} R{} ::: *.png
rather than say ...
8
votes
6answers
11k views
Parallelize a Bash FOR Loop
I have been trying to parallelize the following script, specifically each of the three FOR loop instances, using GNU Parallel but haven't been able to. The 4 commands contained within the FOR loop run ...
2
votes
2answers
810 views
scp does not display output when used with gnu parallel
I have a short script that uses scp to copy files to a number of remote hosts (yes, I know about rdist and rsync; they both fail to work for a few of the hosts - that's not the point here; I'm only ...
0
votes
0answers
132 views
log4cxx/logger.h or libcql/cql.hpp fatal error while doing make -j4
I am running make -j4 and I am getting below exception...
userName@phx5qa01c-4e23:~/build$ make -j4
[ 1%] [ 2%] Building CXX object nda/common/loadtest/CMakeFiles/loadtester.dir/LoadTester.cc.o
...
