Optimising file transfers by improving efficiency from cp to rsync

Transferring files between development and production environments is a critical task in the deployment process. However, I continue to come across multiple approaches that scale from awesome automation using pipelines to the basic of direct command line entry. Where the basic approaches rely on outdated processes that are inefficient and difficult to maintain. Let's explore how to optimise file transfer workflows, moving from rudimentary approaches to more efficient methods using cp and eventually rsync.

If you're not using a pipeline to deploy (and you should be), then how you transfer the files is critical. Pipelines streamline deployment processes by automating tasks, ensuring consistency, and reducing human errors. They allow for testing and validation steps before deployment, improving reliability and speeding up delivery. Using a pipeline also simplifies rollbacks in case issues arise, making deployments safer and more predictable.

I'll walk through deployment scripts, starting with basic file copying, improving through loops, and culminating in a more robust and dynamic solution leveraging rsync. We’ll also discuss how these scripts can be seamlessly integrated into CI/CD pipelines for automated deployments.

What I started with

The initial setup for managing deployments between dev, staging, and production environments was manual process and relied on direct file transfers within the same server. While this approach functioned, it introduced several challenges:

• Scalability issues: The manual process did not scale well as the number of files and directories grew;
• Risk of errors: Human intervention increased the likelihood of mistakes, such as missing files or overwriting changes;
• Security concerns: Keeping all environments on the same server increased the risk of accidental data corruption or unauthorised access affecting multiple environments; and
• Lack of automation: The process required constant monitoring and manual execution, slowing down deployments and increasing operational overhead.

The entry point for managing dev to staging to production was held on the same server. While this setup worked, it lacked scalability and automation. The process relied on the following manual script to transfer files:

cp -R dev/core httpsdocs/
cp -R dev/modules httpsdocs/
cp -R dev/profiles httpsdocs/
cp -R dev/sites httpsdocs/
cp -R dev/themes httpsdocs/
cp -R dev/vendor httpsdocs/
cp dev/autoload.php httpsdocs/
cp dev/composer.json httpsdocs/
cp dev/composer.lock httpsdocs/
cp dev/index.php httpsdocs/
cp dev/robots.txt httpsdocs/
cp dev/update.php httpsdocs/
cp dev/web.config httpsdocs/

cd httpsdocs

chown user:psacln -R core
chown user:psacln -R modules
chown user:psacln -R profiles
chown user:psacln -R themes
chown user:psacln -R sites
chown user:psacln -R vendor
chown user:psacln autoload.php
chown user:psacln composer.lock
chown user:psacln composer.json
chown user:psacln index.php
chown user:psacln robots.txt
chown user:psacln update.php
chown user:psacln web.config

Evolving the basic approach

The original method for transferring files relied on repetitive cp commands, as shown above. While functional, this approach had several limitations:

• Redundant commands: The script was lengthy and difficult to maintain;
• No progress indication: Large transfers left users guessing about the status; and
• Performance issues: Files were copied regardless of whether they had changed, wasting time and resources.

To address these shortcomings, we transitioned to a loop-based approach, simplifying the process and improving maintainability. Additionally, instead of executing commands directly on the server, we consolidated them into a reusable Bash script (deploy_files.sh) for easier execution and updates.

Improving with cp and loops

By using loops, the script became more concise and manageable. Instead of duplicating commands for each directory and file, the loop iterates through lists, making the code easier to update and maintain.

This approach also reduces the likelihood of errors caused by missed or mistyped entries, ensuring all required directories and files are processed consistently.

#!/bin/bash

SOURCE="dev"
TARGET="httpsdocs"

if [ ! -d "$SOURCE" ]; then
  echo "Source directory '$SOURCE' does not exist. Exiting."
  exit 1
fi

if [ ! -d "$TARGET" ]; then
  mkdir "$TARGET"
fi

# Copy directories
for DIR in core modules profiles sites themes vendor; do
  cp -R "$SOURCE/$DIR" "$TARGET/"
done

# Copy files
for FILE in autoload.php composer.json composer.lock index.php robots.txt update.php web.config; do
  cp "$SOURCE/$FILE" "$TARGET/"
done

# Set ownership
chown -R coding:psacln "$TARGET"

This loop-based method saved time, reduced human error, and improved maintainability but still suffered from performance issues due to copying unchanged files.

By using loops, the script became more concise and manageable:

#!/bin/bash

SOURCE="dev"
TARGET="httpsdocs"

if [ ! -d "$SOURCE" ]; then
  echo "Source directory '$SOURCE' does not exist. Exiting."
  exit 1
fi

if [ ! -d "$TARGET" ]; then
  mkdir "$TARGET"
fi

# Copy directories
for DIR in core modules profiles sites themes vendor; do
  cp -R "$SOURCE/$DIR" "$TARGET/"
done

# Copy files
for FILE in autoload.php composer.json composer.lock index.php robots.txt update.php web.config; do
  cp "$SOURCE/$FILE" "$TARGET/"
done

# Set ownership
chown -R coding:psacln "$TARGET"

This loop-based method reduced redundancy but still suffered from performance issues due to copying unchanged files.

Achieving optimal efficiency with rsync

The initial implementation of rsync aimed to improve efficiency by transferring only new or modified files, resulting in significant time savings for larger transfers.

#!/bin/bash

SOURCE="dev/"
TARGET="httpsdocs/"

if [ ! -d "${SOURCE%/}" ]; then
 echo "Source directory '$SOURCE' does not exist. Exiting."
 exit 1
fi

if [ ! -d "${TARGET%/}" ]; then
 mkdir -p "$TARGET"
fi

rsync -a --info=progress2 "$SOURCE" "$TARGET"
chown -R user:psacln "$TARGET"

echo "Script completed successfully!"

This approach significantly improved performance but lacked flexibility, as it copied all directories and files by default.

Refining rsync to copy specific directories and files

The final version of the script supports dynamic source and target directories, allowing flexibility when specifying environments. For example, the script can be executed manually using:

bash deploy_files.sh dev httpsdocs

This command sets the source directory to dev and the target directory to httpdocs. If no arguments are provided, the script defaults to dev/ as the source and httpsdocs/ as the target.

#!/bin/bash

SOURCE=${1:-"dev/"}
TARGET=${2:-"httpsdocs/"}

if [ ! -d "${SOURCE%/}" ]; then
  echo "Source directory '$SOURCE' does not exist. Exiting."
  exit 1
fi

if [ ! -d "${TARGET%/}" ]; then
  mkdir -p "$TARGET"
fi

# Sync specific directories
rsync -a --info=progress2 "$SOURCE/core" "$TARGET/"
rsync -a --info=progress2 "$SOURCE/modules" "$TARGET/"
rsync -a --info=progress2 "$SOURCE/profiles" "$TARGET/"
rsync -a --info=progress2 "$SOURCE/themes" "$TARGET/"
rsync -a --info=progress2 "$SOURCE/vendor" "$TARGET/"

# Sync root files
rsync -a --info=progress2 "$SOURCE/autoload.php" "$TARGET/"
rsync -a --info=progress2 "$SOURCE/composer.json" "$TARGET/"
rsync -a --info=progress2 "$SOURCE/composer.lock" "$TARGET/"
rsync -a --info=progress2 "$SOURCE/index.php" "$TARGET/"
rsync -a --info=progress2 "$SOURCE/robots.txt" "$TARGET/"
rsync -a --info=progress2 "$SOURCE/update.php" "$TARGET/"
rsync -a --info=progress2 "$SOURCE/web.config" "$TARGET/"

chown -R user:psacln "$TARGET"

echo "Script completed successfully!"

Adding to a deployment pipeline

Given you have completed the hard work by creating a bash script, this script can also be integrated into CI/CD pipelines for automated deployments. Here is an example for Azure DevOps:

trigger:
- main
pool:
 vmImage: 'ubuntu-latest'
stages:
- stage: Deploy
 jobs:
 - job: TransferFiles
   steps:
   - checkout: self
   - script: |
       chmod +x deploy_files.sh
       bash deploy_files.sh dev/ httpsdocs/
     displayName: 'Run transfer script'
   - script: |
       echo "Deployment completed successfully!"
     displayName: 'Post-deployment message'

Lessons for deployments

This evolution demonstrates how incremental improvements can lead to more efficient processes. Starting with basic cp commands, transitioning to loops, and ultimately leveraging rsync highlights the importance of refining workflows to meet modern deployment needs.

Key lessons learned include:

• Efficiency through automation: Automating repetitive tasks reduces manual errors and improves reliability;
• Scalability with tools: Tools like rsync enable incremental updates, making the process faster and more scalable;
• Flexibility with dynamic parameters: Supporting dynamic source and target directories provides adaptability for different environments; and
• Integration into CI/CD pipelines: Embedding scripts into deployment pipelines streamlines processes and enforces consistency across environments.

Future recommendations for teams include adopting pipeline-driven deployments as the standard, ensuring testing and validation steps are automated, and continuously reviewing workflows to incorporate advancements in deployment technologies.

This evolution demonstrates how incremental improvements can lead to more efficient processes. From basic cp commands to rsync and pipeline integration, optimising workflows is essential for scalability and reliability.