current community

your communities

more stack exchange communities

Stack Exchange
tour help
Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 2 down vote favorite

I am trying to copy files from testMachineB and testMachineC into testMachineA as I am running my shell script on testMachineA.

If the file is not there in testMachineB, then it should be there in testMachineC for sure. So I will try to copy file from testMachineB first, if it is not there in testMachineB then I will go to testMachineC to copy the same files.

PARTITIONS is the file partition number which I need to copy in testMachineA in directory FOLDER_LOCATION.

#!/bin/bash

readonly FOLDER_LOCATION=/export/home/username/pooking/primary
readonly MACHINES=(testMachineB testMachineC)
PARTITIONS=(0 3 5 7 9 11 13 15 17 19 21 23 25 27 29) # this will have more file numbers around 400

dir1=/data/snapshot/20140317

# delete all the files first
find "$FOLDER_LOCATION" -mindepth 1 -delete
for el in "${PARTITIONS[@]}"
do
    scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 username@${MACHINES[0]}:$dir1/s5_daily_1980_"$el"_200003_5.data $FOLDER_LOCATION/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 username@${MACHINES[1]}:$dir1/s5_daily_1980_"$el"_200003_5.data $FOLDER_LOCATION/.
done

Problem Statement:-

Now what I am trying to do is - Split the PARTITIONS array which contains the partition number in set of five files. So I will copy first set which has 5 files in parallel. Once these five files are done, then I will move to next set which has another five files and download them in parallel again and keep on doing this until all the files are done.

I don't want to download all the files in parallel, just five files at a time.

Is this possible to do using bash shell scripting?

Update:-

Something like this you are suggesting?

echo $$

readonly FOLDER_LOCATION=/export/home/username/pooking/primary
readonly MACHINES=(testMachineB testMachineC)
ELEMENTS=(0 3 5 7 9 11 13 15 17 19 21 23 25 27 29)
LEN_ELEMENTS=${#ELEMENTS[@]}
X=0

dir1=/data/snapshot/20140317

function download() {
    if [[ $X < $LEN_ELEMENTS ]]; then
        (scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 username@${MACHINES[0]}:$dir1/s5_daily_1980_"${ELEMENTS[$X]}"_200003_5.data $FOLDER_LOCATION/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 username@${MACHINES[1]}:$dir1/s5_daily_1980_"${ELEMENTS[$X]}"_200003_5.data $FOLDER_LOCATION/.) && kill -SIGHUP $$ 2>/dev/null &
    fi
}

trap 'X=$((X+1)); download' SIGHUP

# delete old files
find "$FOLDER_LOCATION" -mindepth 1 -delete

# initial loop
for x in {1..5}
do
    download
done

# waiting loop
while [ $X -lt $LEN_ELEMENTS ]
do
    sleep 1
done

Does above looks right? And also, now where do I put my delete command?

share|improve this question
add comment

3 Answers

up vote 2 down vote accepted

Something like this:

# Your variable initialization
readonly FOLDER_LOCATION=/export/home/username/pooking/primary
readonly MACHINES=(testMachineB testMachineC)
PARTITIONS=(0 3 5 7 9 11 13 15 17 19 21 23 25 27 29) # this will have more file numbers around 400

dir1=/data/snapshot/20140317

# delete all the files first
find "$FOLDER_LOCATION" -mindepth 1 -delete

# Bash function to copy a single file based on your script
do_copy() {
  el=$1
  scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/s5_daily_1980_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir1/s5_daily_1980_"$el"_200003_5.data $PRIMARY/.
}

# export -f is needed so GNU Parallel can see the function
export -f do_copy

# Run 5 do_copy in parallel. When one finishes, start another.
# Give them each an argument from PRIMARY_PARTITION
parallel -j 5 do_copy ::: "${PRIMARY_PARTITION[@]}"

To learn more:

10 seconds installation of GNU Parallel:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash
share|improve this answer
    
Thanks for the help. Appreciated your help. Can you add little bit explanation. How will it copy five files at a time. Actually whatever you posted, I was not able to understand at all. –  Webby May 4 at 23:18
    
Thanks for the help. Tell me one more thing, It will download 5 files in parallel? And if they are done, then only it will download next five files? or at a time, it will only download 5 five files in parallel? –  Webby May 5 at 7:36
    
I can recommend again you go through the tutorial. It will (amongst other things) clear up exactly what happens. –  Ole Tange May 5 at 18:16
    
I figured that out after going through the youtube video you posted. Thanks for the help. –  Webby May 5 at 18:18
add comment
up vote 0 down vote

Try with trapping:

#!/bin/bash

echo $$

ELEMENTS=(0 1 2 3 4 5 6)
LEN_ELEMENTS=${#ELEMENTS[@]}
X=0

function download() {
    if [[ $X < $LEN_ELEMENTS ]]; then
        (echo ${ELEMENTS[$X]}) && kill -SIGHUP $$ &
        #^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        # here put your code, important to and it with kill (whole command) 
        # and put & at the end to background this
    fi
}

trap 'X=$((X+1)); download' SIGHUP

# initial loop
for x in {1..5}
do
    X=$((X+1))
    download
done

# waiting loop
while [ $X -lt $LEN_ELEMENTS ]
do
    sleep 1
done

This solution is based on signals sent back to parent process from child ones after they will finish its work, in this case I use SIGHUP signal, and declare it with:

trap download SIGHUP

it means, if you get SIGHUP, execute download. You can bump it by hand, executing kill -SIGHUP <script_pid> Then script is waiting in while loop to signals from children processes.

Caution

This solution is based on signal, which is a two-state value, it means that if many processes will trap at same time (time between traps will less than trap's service time), only first one will be executed.

share|improve this answer
    
Can you add little bit explanation how this is working? Sorry my first language is not shell scripting. And where I am supposed to put my scp command to copy the files in your suggestion? –  Webby May 4 at 19:31
    
I have updated my question to add my script changes in your suggestion. Can you take a look whether this looks ok? And also where I am supposed to add my delete command? If it looks wrong to you, can you add an example for that as well? Appreciated your help. Thanks. –  Webby May 4 at 19:39
    
It's ok, I would add parenthesis around (scp || scp) && kill... to make it work as you expect. What delete you mean? –  fazie May 4 at 19:45
    
Thanks for editing my question. If you see closely s5_daily_1980_"$el"_200003_5.data, I have "$el", I need to replace this with something in your example right? –  Webby May 4 at 19:45
    
By delete, I mean, before copying any files, I delete everything - find "$FOLDER_LOCATION" -mindepth 1 -delete like this. Where should I put this now? –  Webby May 4 at 19:46
show 4 more comments
up vote 0 down vote

For a less sophisticated solution, you could just extract array slices of five elements each and launch your scp commands in the background.

Using the same basic script you posted (but removing the trailing . at the end of the scp command since that is presumably a typo):

#!/bin/bash

#readonly FOLDER_LOCATION=/export/home/username/pooking/primary
readonly FOLDER_LOCATION=/home/terdon/foo/bar
#readonly MACHINES=(testMachineB testMachineC)
readonly MACHINES=(badabing)
PARTITIONS=(0 3 5 7 9 11 13 15 17 19 21 23 25 27 29) # this will have more file numbers around 400

dir1=/data/snapshot/20140317

# delete all the files first
find "$FOLDER_LOCATION" -mindepth 1 -delete

## ${#PARTITIONS[@]} is the number of elements in the arrya.
## Therefore, $lim is set to that number divided by 5. This
## is used later to split the array into five sets of files.
lim=$((${#PARTITIONS[@]}/5))


## The seq command will print the numbers from 0 to
## the length of the array, incrementing by $lim
for i in $(seq 0 $lim ${#PARTITIONS[@]})
do
    ## This will just print the $lim elements of the array starting
    ## from the current value of $i. The sed will convert spaces to commas
    ## so that it is the right format for brace expansion.
    files=$(echo "${PARTITIONS[@]:$i:$lim}" | sed 's/ /,/g')

    ## This launches the scp commands. {"$files"} will expand to the current 
    ## array slice. For example 0,1,2,3. Since this is run in the background,
    ## it will immediately continue on to the next slice. The result will be
    ## 5 scp processes running in parallel, each downloading one file.
    scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 username@${MACHINES[0]}:$dir1/s5_daily_1980_{"$files"}_200003_5.data $FOLDER_LOCATION/ || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 username@${MACHINES[1]}:$dir1/s5_daily_1980_{"$files"}_200003_5.data $FOLDER_LOCATION/ &
done

CAVEAT: In those cases where the length of your array is not cleanly divisible by 5, you might get a 6th scp process launched to get the stranglers. For example:

ARRAY=( $(seq 0 403) )
lim=$((${#ARRAY[@]}/5))
for i in $(seq 0 $lim ${#PARTITIONS[@]}); do echo ${PARTITIONS[@]:$i:$lim}; done

The above will produce 6 lines of output:

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239
240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319
320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399
400 401
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.