Troubleshooting bash script errors - a guide to fixing line endings and hidden characters

If you’ve ever encountered strange errors while running a bash script, you’re not alone. Errors like : not found and Syntax error: end of file unexpected (expecting "then") can be particularly frustrating, especially when the script appears to be perfectly fine. This a situation that I had recently involving such errors and this article shows how I was able to resolve them, focusing on line endings and hidden characters.

Imagine you have a simple bash script designed to check and set the process timeout for Composer, a popular dependency manager for PHP. Here’s the script

#!/bin/bash
# Get the current process-timeout value
timeout=$(composer config --global process-timeout)
# Check if the timeout is set and is a number
if [ -n "$timeout" ] && [ "$timeout" -eq "$timeout" ] 2>/dev/null; then
   if [ "$timeout" -lt 3000 ]; then
       echo "Updating process-timeout to 6000."
       composer config --global process-timeout 6000
   else
       echo "Process-timeout is already set to $timeout."
   fi
else
   echo "Failed to retrieve a valid process-timeout value."
fi

Nothing amazing about this script.  However, when running the script, you encounter a series of confusing error messages:

sh /var/www/html/scripts/check-process-timeout.sh
: not foundml/scripts/check-process-timeout.sh: 2: 
: not foundml/scripts/check-process-timeout.sh: 4: 
: not foundml/scripts/check-process-timeout.sh: 7: 
/var/www/html/scripts/check-process-timeout.sh: 19: Syntax error: end of file unexpected (expecting "then")

What’s going on?

These errors suggest that there may be issues with the script that are not immediately obvious. Here are the likely trouble makers:

• Line endings issue: If the script was edited on a Windows machine, it might contain CRLF (Carriage Return + Line Feed) line endings instead of the LF (Line Feed) used in Unix-based systems. When such scripts are executed in a Unix environment, they can cause errors;
• Hidden characters: Non-visible characters, such as extra spaces or invisible control characters, can cause the shell to misinterpret the script, leading to errors.

Verifying the script structure

Before diving into solutions, it’s important to verify that the script is logically correct. Upon reviewing the script, we found that all if statements are properly closed with corresponding fi statements. Here’s a breakdown of the if blocks:

if [ -n "$timeout" ] && [ "$timeout" -eq "$timeout" ] 2>/dev/null; then
   # Inner if block
   if [ "$timeout" -lt 3000 ]; then
       # Update process-timeout
   else
       # Process-timeout is already set
   fi
else
   # Failed to retrieve a valid process-timeout value
fi

Since there are no missing fi statements, the issue must be related to the environment where the script was edited or executed.

Fixing line endings

For Unix/Linux systems

If you’re working in a Unix-like environment, the simplest way to fix line ending issues is by using the dos2unix command:

dos2unix /var/www/html/scripts/check-process-timeout.sh

This command converts Windows-style line endings (CRLF) to Unix-style line endings (LF), resolving any issues related to line endings.

For Windows 10 Users

Windows 10 does not include the dos2unix command by default, but you can achieve the same result using text editors like Notepad++ or Visual Studio Code.

Using Notepad++

Open the script in Notepad++.
Go to Edit > EOL Conversion > UNIX (LF).
Save the file.
Using Visual Studio Code:

Using Visual Studio Code

Open the script in Visual Studio Code.
In the bottom-right corner of the window, click where it says CRLF (if it's using Windows line endings).
Change it to LF.
Save the file.

These steps convert the line endings to the appropriate format for Unix environments.

Changing line endings for an entire project in VS Code

If you’re working on a larger project and want to ensure that all files in a directory or project use the correct line endings, you can do so in Visual Studio Code.

Change default end-of-line setting in VS Code

To ensure that all new files and those you open in VS Code use LF (Unix-style) line endings.

1. Open VS Code
2. Open Settings:
   1. Click on the gear icon in the bottom-left corner and select "Settings," or use the shortcut Ctrl + ,.
3. Search for "End of Line" in the search bar
4. Change the Setting:
   1. Look for the setting "Files: Eol" and change it to \n. This sets the default end-of-line sequence to LF for all files.
   2. Alternatively, you can directly add the following to your settings.json file:

"files.eol": "\n"

Convert line endings for all files in a directory

While VS Code does not natively support batch conversion of line endings for all files in a directory, you can use an extension or a command-line approach.

Using an Extension:

1. Install the "Change All End Of Line Sequence" Extension:
   1. Go to the Extensions view (Ctrl + Shift + X) and search for "Change All End Of Line Sequence."
   2. Install the extension
2. Run the Extension:
   1. Open the Command Palette (Ctrl + Shift + P)
   2. Search for "Change All End Of Line Sequence" and select it
   3. Choose LF from the options.

Using Command-Line:

Otherwise using the terminal, you can use the find and sed commands on Unix-like systems to convert all files in a directory:

find /path/to/your/project -type f -exec sed -i 's/\r$//' {} \;

Replace /path/to/your/project with the actual path to your project directory. This command will recursively find all files and convert their line endings to LF.

Apply the Change Globally with .editorconfig

To ensure consistency across your project, add an .editorconfig file at the root of your project:

• Create/Edit .editorconfig
• Create a file named .editorconfig in the root of your project
• Add the following configuration

root = true
[*]
end_of_line = lf
charset = utf-8
insert_final_newline = true
trim_trailing_whitespace = true

• Save the file

This ensures that all files in your project adhere to the LF line ending format, regardless of individual settings.

Checking for hidden characters

Even after fixing line endings, hidden characters might still be an issue. You can inspect and remove these using a text editor or the sed command:

sed -i 's/\r$//' /var/www/html/scripts/check-process-timeout.sh

This command removes any trailing \r characters, which are common in files edited on Windows.

The wrap

Bash script errors like the ones we explored can often be traced back to environment-specific issues, particularly when switching between Windows and Unix-like systems. By understanding and addressing line endings and hidden characters, you can avoid these frustrating problems.