Shell scripting, reading a list from tt file, use as variable, use the variable to run python scripts in a foor loop

Question

I am trying to run a shell script where it reads the variables from a txt file, uses that variable in a for loop to run python file.

1. The code reads a txt file where at least there are 100 variable such as var1, var2, var3....var 100

2. The code changes the directory (Rep1, Rep2) to read another file from each directory RepOut_Rep1.out

3. The code runs a python code where it takes the entries from different directories

   python /home/PhytonFiles/phytonFold.py  -s /home/mySFiles/$var -o $RepOut_Rep1.out -n summaryFile.txt
   

I wrote the following code but I am afraid it did not work

input="MyList.txt"
while IFS= read -r file; 
do
# printf '%s\n' "$file"
  
# run the script for the following directories (Rep1, Rep2. ...)
  for f in Rep1 Rep2
  do
  cd ${f}

#  pull the output file 
  outFile=RepOut_${f}.out

# create a summary folder at the end of the run
  summaryFile=summary_${f}_$file

# run the python file, get the inputs from /home/mySFiles/ with the variable $file
  phyton /home/PhytonFiles/phytonFile.py  -s /home/mySFiles/$file -o $outFile -n $summaryFile 

done 
done < "$input"

I am not sure if my use of variables are correct in the python run line. Where do I make mistakes?

Thank you for your help.

Bircan

Answer 1

There are actually a few things that are preventing this from doing what you want.

1. The most important is that on line 9 you cd into Rep1, and then you can't cd into Rep2 when the for loop continues because you're still in Rep1

so if you're folder structure is

./
  ./Rep1
  ./Rep2

you need to then cd back out into the parent folder the line after you run the python script. But to be honest it's unclear why you're using the cd command at all just write out the whole file path in the for loop.

2. You've also spelled python wrong so that python line probably was never executed.

3. if you mean a file of variables that looks like this:

VAR1=a VAR2=b VAR3=c ...

then you will want to source that file not read from it, but I'm guessing based on the script that that file is not actually a file of variables as in bash variables but instead a folder of filenames like

file1.txt
file2.txt
file3.txt
...

If that's the case, then you'll want to make sure you're always putting $file in quotes because you don't know if any of the file names have spaces, and bash print something like this in the python line if there are any spaces:

phyton /home/PhytonFiles/phytonFile.py  -s /home/mySFiles/some file name.txt -o $outFile -n $summaryFile 

This will make python think that there are these options:

-s /home/mySFiles/some
-o $outFile
-n $summaryFile

and these arguments

file
name.txt

Here's a version of the script with the changes I just described :)

#!/bin/bash

input="MyList.txt"
while IFS= read -r file;
do
# printf '%s\n' "$file"

  # run the script for the following directories (Rep1, Rep2. ...)
  for f in Rep1 Rep2
  do

  # pull the output file
  outFile="RepOut_${f}.out"
  printf "\noutFile:\t%s\n" "$outFile"

  # create a summary folder at the end of the run
  summaryFile="summary_${f}_$file"
  printf "summaryFile:\t%s\n" "$summaryFile"

  # run the python file, get the inputs from /home/mySFiles/ with the variable $file
  echo python /home/PhytonFiles/phytonFile.py  -s "/home/mySFiles/$file" -o "$outFile" -n "$summaryFile"
  done
done < "$input"

I also included a nice trick with these kinds of problems, I've put the word "echo" at the front of the Python line. Go ahead and run it the way it is and it will let you preview what that line is going to look like for each iteration.

Once it looks correct to you, remove the "echo" from the beginning and run it again and it will actually execute the code.

Best of luck.