current community

your communities

more stack exchange communities

Stack Exchange
tour help
Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 1 down vote favorite

I'm executing 60 scripts with GNU parallel(they all have wget commands in there)but I have noticed that after a few hours execution will slow down a bit. What could be causing this?

I'm executing parallel with this command: parallel -j 60 < list where "list" is just a file with directories to 60 scripts.

I'm on a CentOS 6.5 machine.

share|improve this question
1  
Since you use wget it might be highly related to network issues for the slower execution of the scripts. –  Ramesh Apr 8 '14 at 22:47
    
If I where to stop the execution at any given time and restart it, it would start going fast again, what could I do to see if this is related to a network problem? –  Dominique Apr 8 '14 at 22:50
    
Recently I was extracting billions of rows of data from the internet. From my understanding, when you start the execution not all the 60 processes clog the network traffic initially. However, as the time frame increases, the network traffic increases and the bandwidth is fully utilized. Also, it might be related to the CPU usage. –  Ramesh Apr 8 '14 at 22:54
    
Though theory wise it is possible to run multiple threads, it is highly machine dependent. If your machine is single core and you run 60 processes, not all 60 processes will take place in parallel. I am not sure of this concept but this is definitely another reason for the slowness. –  Ramesh Apr 8 '14 at 22:55
    
@Ramesh in htop I see minimal CPU usage and network usage remains constant. This VM(hosted on kvm)has 8 threads and 24GBs of RAM. –  Dominique Apr 8 '14 at 23:29

1 Answer 1

up vote 1 down vote accepted

From Understanding the Linux Kernel:

In Linux, process priority is dynamic. The scheduler keeps track of what processes are doing and adjusts their priorities periodically; in this way, processes that have been denied the use of the CPU for a long time interval are boosted by dynamically increasing their priority. Correspondingly, processes running for a long time are penalized by decreasing their priority.

share|improve this answer
    
Is there a way I can force a high priority? –  Dominique Apr 8 '14 at 22:55
    
Use the renice command. –  Kyle Jones Apr 8 '14 at 22:57
    
Will the renice command affect the commands invoked by the bash scripts and parallel? –  Dominique Apr 8 '14 at 23:31
    
If you renice the parent process, all future spawned children will share that niceness value. –  Kyle Jones Apr 9 '14 at 1:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.