Performing tasks in parallel, to make use of multiple processors
2
votes
3answers
105 views
Execute command on multiple files matching a pattern in parallel
Let's say I have a command accepting a single argument which is a file path:
mycommand myfile.txt
Now I want to execute this command over multiple files in parallel, more specifically, file ...
2
votes
1answer
31 views
multicore CLI commands question
I have a collection of 150 CLI commands to execute in a file, on my 12 core machine. Running all 150 tasks in the background bogs the system down and I have found by experience that having 12-18 tasks ...
1
vote
3answers
50 views
bash multi-threads help require
I have a list of IPs and I need to check them for opened ports using nmap.
So far, my script is like this:
#!/bin/bash
filename="$1"
port="$2"
echo "STARTING NMAP"
while IFS= read -r line
do
nmap ...
0
votes
1answer
17 views
parallel execution of offlineimap or abuse pythonfile/postsynchook to load second file for account settings
Why do i need that?
I've set up my dotfiles to include my whole email configuration, but i don't want to share my my private email account settings, so i excluded them.
=> For my mutt/offlineimap i ...
3
votes
1answer
38 views
parallelize shell script
I have a following line:
for length in "$(ls $OUT_SAMPLE)"
do $CODES/transform_into_line2.rb -c $OUT_SAMPLE -p 0 -f $length &
done
So,it should parallelize the for loop but somehow it still ...
16
votes
5answers
739 views
How to run scripts in parallel on a remote machine?
I can ssh into a remote machine that has 64 cores. Lets say I need to run 640 shell scripts in parallel on this machine. How do I do this?
I can see splitting the 640 scripts into 64 groups each of ...
1
vote
2answers
63 views
How do I quickly transfer a large archive (~16GB)
How do I quickly transfer a large archive (~16GB) from one mount point to another on 80 servers? The mount points are on separate partitions. I tried ansible (with the -f 100 option) and a bash script ...
3
votes
3answers
51 views
query r.e running for loop scripts in parallel
I have the following in a shell script:
for file in $local_dir/myfile.log.*;
do
file_name=$(basename $file);
server_name=$(echo $file_name | cut -f 3 -d '.');
...
3
votes
1answer
47 views
Where/when are exit statuses returned for background jobs in bash?
I'm writing a bash script which has many parts, but I want built in resiliency and parallel processing as much as possible. An example is below:
while true
do
var=$(curl someurl) || continue
...
2
votes
2answers
99 views
Uncompressed .lzo files in parallel and then delete the original .lzo files
So I have .lzo files in /test01/primary folder which I need to uncompress and then delete all the .lzo files. Same thing I need to do in /test02/secondary folder as well. I will have around 150.lzo ...
0
votes
1answer
96 views
Running two commands at the same time?
Here is the nature of my problem:
I'm using my raspberry pi to run a wikipedia server, so I'd like to make a launcher (desktop icon) that starts up the wikipedia server and opens the web browser to ...
0
votes
3answers
74 views
New signal arrives while execution is in a signal handler, how to decide which is first?
I am working on a tool handling a massive amount of signals (with different signums) with sigaction().
I need to handle the case, if a new signal is coming, while the previous was in a signal ...
3
votes
6answers
407 views
How to run no more than n parallel subshells
I am trying to run subscripts from a main script, but I want to make sure than no more than n subscripts run at the same time.
The following simplified example illustrates.
Each subscript creates a ...
0
votes
0answers
42 views
LSF bsub implimentation of LUT
We have an implementation of LSF (bsub) on our HPC system. I want to run a model several times with various input parameters. I will have a table either csv or other format with the required ...
1
vote
1answer
104 views
tar in parallel [duplicate]
I have a directory with several hundred sub-directories. I want to tar and compress each sub-directory and name the resulting file <currentDirName>.tar.bz2.
I have done:
find ./dir -type f ...
0
votes
1answer
314 views
Running multiple commands at the same time
I want to run two commands on terminal on my virtual machine at the same time.
I have this as of now:
sudo ptpd -c -g -b eth1 -h -D; sudo tcpdump -nni eth1 -e icmp[icmptype] == 8 -w capmasv6.pcap
...
3
votes
2answers
212 views
How to run x instances of a script parallel?
I have script I'd always like to run 'x' instances parallel.
The code looks a like that:
for A in
do
for B in
do
(script1.sh $A $B;script2.sh $A $B) &
done #B
done #A
The scripts ...
3
votes
3answers
567 views
Linux & Linux-Software: Advantages of a multi core processor [closed]
I plan to get a new notebook and try to find out if a quad-core processor gives me any advantages over a regular dual-core machine. I use common Linux Distributions (Ubuntu, Arch etc.) and mostly ...
3
votes
2answers
363 views
How to determine the maximum number to pass to make -j option?
I want to compile as fast as possible. Go figure. And would like to automate the choice of the number following the -j option. How can I programmatically choose that value, e.g. in a shell script?
...
1
vote
1answer
76 views
In a Makefile, how can I begin processing a file before it has finished building?
I'm using make to automate a project that generates lots of data, then processes it, with recipes roughly like this:
processed-data: data
data-processing-program $^ > $@
data:
output-data ...
1
vote
2answers
549 views
Spawn multiple processes with a single command
Is there a way to create say 10 instances of a process (for example yes) with a single command?
$instantiate 10 yes
3
votes
4answers
159 views
How to maintain a specific number of processes running?
I'm trying to write a shell script for converting a lot of .wav files to .mp3. I use the "find" command to find them, then the "lame" command to convert. I can use a simple for loop for this but I ...
11
votes
3answers
3k views
How to start multi-threaded grep in terminal?
I have a folder which has 250+ files of 2 GB each. I need to search for a string/pattern in those files and output the result in an output file. I know I can run the following command, but it is too ...
8
votes
3answers
353 views
correct xargs parallel usage
I am using xargs to call a python script to process about 30 million small files. I hope to use xargs to parallelize the process. The command I am using is:
find ./data -name "*.json" -print0 |
...
3
votes
3answers
490 views
How to run multiple scripts in parallel
I have one bash script which calls the same perl scrips in a serial way. The bash script is used to collect the overall results, while the perl script collects the results of my simulations for the ...
2
votes
3answers
209 views
Can you make a process pool with shell scripts?
Say I have a great number of jobs (dozens or hundreds) that need doing, but they're CPU intensive and only a few can be run at once. Is there an easy way to run X jobs at once and start a new one ...
0
votes
1answer
42 views
Download data in parallel [closed]
I have to download 20G of data using ftp .Can I do this in parallel(gnu?)
Here are some of the links for the data.
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR030/ERR030893/ERR030893.fastq.gz
...
0
votes
2answers
1k views
Run two scripts at the same time
I want to run two scripts at the same time. How exactly do I do this?
If I have a variable called foo in script1 and change its value to 5, and if I'm using a variable with the same name in script2 ...
-1
votes
1answer
240 views
Run a server and client in a makefile [closed]
I'm coding a socket file server in C++, and I can't figure out how to put proper unit testing into my makefile. My problem is as follows:
The server, when started, spits out its port number to ...
10
votes
7answers
25k views
Calling multiple bash scripts and running them in parallel, not in sequence
Suppose that I have three (or more) bash scripts: script1.sh, script2.sh, and script3.sh. I would like to call all three of these scripts and run them in parallel. One way to do this is to just ...
1
vote
1answer
500 views
How to find max parallel execution limit?
is there any limit for parallel execution? if yes, how to find out the maximum limit?
I am creating a script which create a string of scripts concatenated by '&' and uses eval to execute them all ...
2
votes
0answers
103 views
Copy folders at once
I want to copy a large number of folders at once not one by one. Is it correct to use the following code ? Is it helping to run the command cp for all the folders i at once?
#!/bin/bash
foreach i in ...
17
votes
6answers
3k views
Simultaneously calculate multiple digests (md5, sha256)?
Under the assumption that disk I/O and free RAM is a bottleneck (while CPU time is not the limitation), does a tool exist that can calculate multiple message digests at once?
I am particularly ...
5
votes
2answers
1k views
How to parallelize dd?
I'm currently having trouble with dd invoked with a sparse file as input (if) and a file as output (of) with conv=sparse. dd seems to be using one core of the CPU (Intel(R) Core(TM) i7-3632QM CPU @ ...
1
vote
0answers
118 views
Microcomputer cluster for stress testing
I'm looking to build a microcomputer cluster at work for stress testing one of our applications, and while I have a little financial leeway, I would like to keep the cost low and the computing ...
1
vote
4answers
849 views
How to run a script in multiple instances ? ( Ubuntu server )
How can I run a script in multiple instances in Ubuntu server?
For example, I have a long running script named scanner.sh placed in /usr/bin/, which I wish to run the script in 10 parallel ...
36
votes
1answer
2k views
How do I distribute a large download over multiple computers?
I need to download a large file (1GB). I also have access to multiple computers running Linux, but each is limited to a 50kB/s download speed by an admin policy.
How do I distribute downloading this ...
2
votes
0answers
552 views
How to pass multiple commands via pssh and multiple xterms
I'm attempting to open multiple xterms and run a command as an SAP user via sudo using pssh. So far, I'm able to run PSSH to a file of servers with no check for keys, open every xterm in the servers ...
6
votes
1answer
9k views
Multi-Threading/Forking in a bash script
I have written a bash script which is in following format:
#!/bin/bash
start=$(date +%s)
inFile="input.txt"
outFile="output.csv"
rm -f $inFile $outFile
while read line
do
-- Block of Commands
...
3
votes
2answers
815 views
split a file, pass each piece as a param to a script, run each script in parallel
I have a words.txt with 10000 words (one to a line). I have 5,000 documents. I want to see which documents contain which of those words (with a regex pattern around the word). I have a script.sh that ...
4
votes
1answer
241 views
Multi-machine tool in the spirit of moreutils' `parallel`?
parallel from moreutils is a great tool for, among other things, distributing m independent tasks evenly over n CPUs. Does anybody know of a tool that accomplishes the same thing for multiple ...
