current community

your communities

more stack exchange communities

Stack Exchange
tour help
Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 1 down vote favorite

I am trying to copy files from machineB and machineC into machineA as I am running my below shell script on machineA.

If the files is not there in machineB then it should be there in machineC for sure so I will try copying the files from machineB first, if it is not there in machineB then I will try copying the same files from machineC.

I am copying the files in parallel using GNU Parallel library and it is working fine. Currently I am copying two files in parallel.

Earlier, I was copying the PRIMARY_PARTITION files in PRIMARY folder using GNU parallel and once that is done, then onnly I was copying the SECONDARY_PARTITION files in SECONDARY folder using same GNU parallel so it is sequential as of now w.r.t PRIMARY and SECONDARY folder.

Now I decided to copy files in PRIMARY and SECONDARY folder simultaneously. Meaning, I will copy two files in PRIMARY folder along with two files in SECONDARY folder simultaneously.

Below is my shell script which I have -

#!/bin/bash

export PRIMARY=/test01/primary
export SECONDARY=/test02/secondary
readonly FILERS_LOCATION=(machineB machineC)
export FILERS_LOCATION_1=${FILERS_LOCATION[0]}
export FILERS_LOCATION_2=${FILERS_LOCATION[1]}
PRIMARY_PARTITION=(550 274 2 546 278) # this will have more file numbers
SECONDARY_PARTITION=(1643 1103 1372 1096 1369 1568) # this will have more file numbers

export dir3=/testing/snapshot/20140103

find "$PRIMARY" -mindepth 1 -delete
find "$SECONDARY" -mindepth 1 -delete

do_CopyInPrimary() {
  el=$1
  scp david@$FILERS_LOCATION_1:$dir3/new_weekly_2014_"$el"_200003_5.data $PRIMARY/. || scp david@$FILERS_LOCATION_2:$dir3/new_weekly_2014_"$el"_200003_5.data $PRIMARY/.
}
export -f do_CopyInPrimary

do_CopyInSecondary() {
  el=$1
  scp david@$FILERS_LOCATION_1:$dir3/new_weekly_2014_"$el"_200003_5.data $SECONDARY/. || scp david@$FILERS_LOCATION_2:$dir3/new_weekly_2014_"$el"_200003_5.data $SECONDARY/.
}
export -f do_CopyInSecondary


parallel -j 2 do_CopyInPrimary ::: "${PRIMARY_PARTITION[@]}" &
parallel -j 2 do_CopyInSecondary ::: "${SECONDARY_PARTITION[@]}" &
wait

echo "All files copied."

Problem Statement:-

With the above script at some point I am getting this exception -

ssh_exchange_identification: Connection closed by remote host
ssh_exchange_identification: Connection closed by remote host
ssh_exchange_identification: Connection closed by remote host

Is there any better way of doing the same thing as the way I am doing currently? I guess, I can still use GNU Parallel to make it work?

share|improve this question

1 Answer 1

up vote 3 down vote accepted

The error is typically caused by too many ssh/scp starting at the same time. That is a bit odd as you at most run 4. That leads me to believe /etc/ssh/sshd_config:MaxStartups and MaxSessions on $FILERS_LOCATION_1+2 is set too low.

Luckily we can ask GNU Parallel to retry if a command fails:

do_Copy() {
  el=$1
  PRIMSEC=$2
  scp david@$FILERS_LOCATION_1:$dir3/new_weekly_2014_"$el"_200003_5.data $PRIMSEC/. || scp david@$FILERS_LOCATION_2:$dir3/new_weekly_2014_"$el"_200003_5.data $PRIMSEC/.
}
export -f do_Copy

parallel --retries 10 -j 2 do_Copy {} $PRIMARY ::: "${PRIMARY_PARTITION[@]}" &
parallel --retries 10 -j 2 do_Copy {} $SECONDARY ::: "${SECONDARY_PARTITION[@]}" &
wait

echo "All files copied."
share|improve this answer
    
Thanks for suggestion again. Can you explain little bit how $1 and $2 is getting used here? And how does it know what is $1 and what is $2 And what does ::: this means? Just little bit explanation will help me a lot. –  lining May 21 '14 at 22:59
    
Start by going through the tutorial: man parallel_tutorial –  Ole Tange May 22 '14 at 9:49

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.