Take the 2-minute tour ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.

I have a simple Python script which will execute a shell script using subprocess module in Python.

Below is my Python shell script which is calling testing.sh shell script and it works fine.

import os
import json
import subprocess

jsonData = '{"pp": [0,3,5,7,9]}'
jj = json.loads(jsonData)

os.putenv( 'jj3', ' '.join( str(v) for v in jj['pp']  ) )

print "start"
proc = subprocess.Popen('testing.sh', stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdout, stderr) = proc.communicate()
if stderr:
   print "Shell script gave some error"
   print stderr
else:
   print stdout
   print "end" # Shell script ran fine.

And below is my testing.sh shell script -

#!/bin/bash

dir1=some_directory
dir2=some_directory

length1=some_number
length2=some_number

if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]
then
    for el in $jj3
    do
        scp david@machineB:/data/be_t1_snapshot/20131215/t1_"$el"_5.data /data01/primary/. || scp david@machineC:/data/be_t1_snapshot/20131215/t1_"$el"_5.data /data01/primary/.
    done        
fi

What my above shell script does is, it will copy the files from machineB OR machineC to machineA. If the files are not there in machineB then it should be there in machineC always. So it will try to copy the files from machineB to machineA but if the files are not there in machineB then it will try to copy the files from machineC to machineA.

Now my above Python script (which I am running from machineA) tries to execute my above shell script and see whether my script got executed successfully or not. As I am storing the stderr of my shell script and stdout as well.

Problem Statement:-

With the above approach there is one problem that I am seeing. As I mentioned if the files are not there in machineB, then it will try to copy the files from machineC to machineA. So whenever I run my Python script which calls my shell script, what happens is that, some files are not there in machineB so it throws some exception and then it tries copying the files from machineC but when the call comes back to Python script, it always go inside if stderr: block as there was an error while copying the files from machineB (which is not what I want) and end print statement doesn't gets printed out.

So the question is if the files are not there in machineB, it will try copying the files from machineC and if the file is there is machineC and got successfully copied to machineA without any error, then I want to call that as a success instead of failure. In current scenario what is happening, if the files are not there in machineB and got successfully copied from machineC then it still counts as a failure and end print statement doesn't gets printed out.

I also want to see whether my shell script has any problem while executing from Python. If yes, then I don't want to print end statement.

How should I overcome this problem?

UPDATED:-

What will happen with the below shell script? As the first for loop will failed because david is not a linux command, but my scp command works fine. So still I will see status code as not equal to 0?

for i in $( david ); do
    echo item: $i
done

dir1=some_directory
dir2=some_directory

length1=some_number
length2=some_number

if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]
then
    for el in $jj3
    do
        scp david@machineB:/data/be_t1_snapshot/20131215/t1_"$el"_5.data /data01/primary/. || scp david@machineC:/data/be_t1_snapshot/20131215/t1_"$el"_5.data /data01/primary/.
    done        
fi
share|improve this question
    
Why not just use rsync? –  Elliott Frisch Jan 1 at 5:18
    
Never worked before so need to spend some time to understand and setup in my production machines. I have worked with SCP before so was aware how to do it. And I am not sure how easy it is for my current problem definition. As I need to execute this shell script from Python. –  SSH Jan 1 at 5:20
    
Seriously. I would use rsync over ssh, I would not pick scp to keep files in sync across three machines. –  Elliott Frisch Jan 1 at 5:23
    
I will try to read more about rsync and see how easily I can incorporate it with my current problem. But for now, I will try to look for solution with what I have. –  SSH Jan 1 at 5:24
add comment

1 Answer

A process which worked correctly can still write to stderr, so it is wrong to check for problems using if stderr:

Instead you should check the processes exit code. If it is not equal to zero, there was an error, e.g:

[...]
if proc.returncode != 0:
    print "Shell script gave some error"
[...]
share|improve this answer
    
Thanks for suggestion. I updated my question with another simple shell script. What will happen in that scenario? I will see status code not equal to zero if any of the command in my shell script failed or only the last command? –  SSH Jan 1 at 5:43
    
@SSH By default shell scripts tend to ignore errors and happily carry on. In bash, you can use set -o errexit so the script will exit on error (with a non-zero exit code) –  dbr Jan 1 at 5:48
    
But if I am using set -o errexit in my shell script. And if it tries to copy the file from machineB and if the files is not there in machineB so it throws an exception but that file will be there in machineC so it will copy the files from machineC. But if I am using set -o errexit then it will stop the shell script as soon as it tries to copy the file from machineB which is also not what I want? –  SSH Jan 1 at 5:54
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.