Shell script works when saved with nano but not when saved with Notepad++

Question

When I copy my bash script across from Notepad++ into a new file with nano editor inside SSH and save. It runs fine. (sh ./install).

But if I save the file (Exact same contents), upload to my web server, download it using Wget on the same machine. I get syntax errors. I've checked the encoding and they seem to be the same. I have since used a plethora of character encoding to see if it will resolve the issue. I am also setting the file to executable once I download it using wget!

The file runs fine and I have zero errors when copy and pasting using nano. Any idea what this may be?

Answer 1

I would be willing to bet that the problem is related to line endings. You're probably going through a non-*nix machine somewhere along the line. I also ran into a problem once where apache (running on Linux) was adding windows style line endings to uploaded text files so you might be seeing something similar.

To test, take the file you downloaded and pass it through od. If it's a long file, just grab the first few lines:

head script.sh | od -c

Look through the output and check if you have anything like this:

f   o   o  \r  \n

The \r is a carriage return and on Windows, lines are ended with \r\n as opposed to \n on *nix. If it turns out that this was indeed the problem, you can fix your file by removing the carriage returns:

sed -i 's/\r//g' script.sh

Answer 2

As @graeme has astutely pointed out since you have the script in both forms on the server you could perform a simple diff to determine what's different between the working version and the problematic version.

$ diff working.sh broken.sh

You can also do a side-by-side diff like this:

$ diff -y working.sh broken.sh

If the script isn't working because of some sort of typo you can often times detect these by adding the -x switch to bash, which causes it to be verbose.

$ bash -x broken.sh

You can also incorporate this switch into the shebang (#!/bin/bash) at the top of your scripts like this:

#!/bin/bash -x

Line endings

This is often the issue when moving files from Windows to Unix/Linux systems. The issue has to do with how the ends of lines are denoted on the 2 platforms. You can read more about it here on Wikipedia, titled: Newlines.

make a sample file

$ echo -e "This is a file.\nThat I made on Unix.\n" > unixfile.txt

As @terdon has described in his answer, you can use sed to strip these out, you can also often use a tool called dos2unix to do the same thing too. You can use it in either 1 of 2 ways:

$ dos2unix unixfile.txt

or if you do not want to overwrite the existing file:

$ dos2unix -n oldfile.txt newfile.txt

When you use the above diff I mentioned earlier you'll get output like this when you compare these 2 files:

$ diff -y unixfile.txt winfile.txt 
This is a couple                            |   This is a couple
of lines of sample                          |   of lines of sample
text.                                       |   text.

You won't be able to discern the differences, just that they're there. Again @terdon's answer shows one method for routing out the issue using od. You can of course use a variety of ways to figure out what's up.

Using vim

with the file cmd.

$ file unixfile.txt
winfile.txt: ASCII text 

$ file winfile.txt 
unixfile.txt: ASCII text, with CRLF line terminators

The above is highlighting the issue, that the file from Windows has CRLF (aka. Carriage Return + Linefeed characters at the end of the lines). These characters are 0x0D & 0x0A in hex, again see the Wikipedia article about Newlines if you want to know more about it.

You can also use vim to see the issue too:

$ vim winfile.txt

Here's a little sequence that shows what to do in vim to see the issue. The CRLF characters typically show up in Unix as ^M, that's a Ctrl+M.

                                       

The sequence is showing me re-opening the file, winfile.txt as a formatted Unix file (:e ++ff=unix). This tells vim not to auto-detect that the file is formatted for Windows, and so it will display the ^M line termination characters.