Skip to main content

Recently I have been investigating one of my Linux server's speed using the copy [cp] command.  This came about due to 'upgrading' the server from Centos 6 to 7.  Once I had migrated a domain, I ran the following command

yes | cp -rf 'stg/core' 'tmpdir'

To my surprise, the first run of this command took over 38 minutes to complete.  Crap!  Yes a simple cp -a command is a very painful and slow process.  Read more about the pain and times under Centos 7 in this article Allegedly a server upgrade to Centos 7.  Is this a bit of Centos 7 love or a server configuration gone horribly wrong or time to change commands or a combination of all or some of these options.

After spending quite a bit of time playing around with cp command, I wanted to bench mark against another copy command.  Step rsync.

 

Rsync to the rescue?

As the name indicates, rsync command is used to sync (or copy) files and directories.  An important feature of rsync is that it works on “delta transfer algorithm”, meaning it will only sync or copy the changes from source to destination instead of copying the whole file.  Ultimately reducing the amount of data sent over network.

command description
-v, –verbose verbose output
-q, –quiet suppress message output
-a, –archive archive files and directory while synchronizing ( -a equal to following options -rlptgoD)
-r, --recursive sync files and directories recursively
-b, --backup take the backup during synchronization
-u, --update don’t copy the files from source to destination if destination files are newer
-l, --links copy symlinks as symlinks during the sync
-n, --dry-run perform a trial run without synchronization
-e, --rsh=COMMAND mention the remote shell to use in rsync
-z, --compress compress file data during the transfer
-h, --human-readable display the output numbers in a human-readable format
-P, --progress show the sync progress during transfer
-S, --sparse handle sparse files efficiently

 

Example commands
command description example
rsync -zavh Copy or Sync directory locally rsync -zavh /home/accounts /work
rsync -zarvh Copy or sync files and directories from local to remote system rsync -zarvh /home/accounts user@192.54.189.22/opt/path
rsync -zarvh --progress Same as above, however this show synchronisation progress rsync -zarvh --progress /home/accounts user@192.54.189.22/opt/path

 

What command did I use to copy a directory with 109MB file size

rsync -arv httpdocs/core tmpdir

 

Post the initial run, subsequent runs will have the -u prompt added

rsync -aurv httpdocs/core tmpdir

 

For a complete list of rsync commands, run the command

rsync -h

 

Times

How did rsync compare to cp?

run cp rsync
1 7:09 0:08
2 0:01 0:00:10

 

Related articles

Andrew Fletcher11 Aug 2022

Using bash script to overwrite files and directories

I had generated a backup of key directories on the server - see Create a Ubuntu backup shell script.  With a backup in hand, now it was time to test how to unpack a backup and overwrite the directories. First I unpacked the backup to a temporary directory tmp-cc. Now I wanted to...
Andrew Fletcher11 Jul 2022

How to synchronize two remote Git repositories

Currently, I have a situation where I have two repositories.  The first is where I've held the code since the beginning.  The second has come on board recently and is the client repo.  However, rather than shut the original down, I want to keep both running with the same code....
Andrew Fletcher09 Jul 2022

Creating a bash script and executing it on OSX

Creating a bash script Bash scripts are files containing code that tell your computer to do something.  In this instance, I want to rename a batch files in a directory.  Such as changing:   37351001 - BlueSpotted.jpg 37351008 -SpangledEmporer _ 0133A.jpg 37353003...