The Node

Posts Tagged ‘rsync’

rsync is a command line tool commonly used to sync two folders on the same server. An example usage would be something like: -

rsync -r -t -v /source-folder /destination-folder

This overwrites existing files on the destination-folder having the same name as files in source-folder. The -r option allows for recursion into subfolders. If you want to preserve newer existing files from overwriting, you can type:-

rsync -r -t -v -u /source-folder /destination-folder

or if you want to backup existing files before overwriting them

rsync -r -t -v -b /source-folder /destination-folder

Another useful switch is –delete-excluded: -

rsync -r -t -v --delete-excluded /source-folder /destination-folder

This causes rsync to delete files on the target-folder that have been excluded on the source-folder. This is useful when you decide that certain files are being backed up unnecessarily.
However, for really secure backups, you’ll want to backup files onto a different server entirely. You can do this still using rsync and SSH. You obviously must have rsync and SSH installed on both your local and your target remote machine. rsync can use SSH as a secure transport agent for the files.

Make sure rsync is installed by opening a terminal session and typing:-

rsync --version

on each machine (i.e. both the target for the backups and the source). If you receive “command not found” or a similar message, you’ll need to download and install rsync. This can be done on Fedora with: -

yum install rsync

and on Ubuntu with: -

sudo apt-get install rsync

Assuming SSHd and rsync are both installed and you can ssh into your remote backup machine, you’ll want to automate the process of backing up files via rsync over SSH. Ordinarily, you’d need to enter a password to ssh into a remote machine which obviously isn’t very useful for automated backups. Therefore, you’ll need to use the public key encryption aspects of SSH to allow secure communication between both machines. On the local machine enter: -

ssh-keygen -t dsa -b 1024 -f ~/rsync-key

When the machine asks for a passphrase, it’s important to simply press enter. Two key files have been created, a public key and a private key (you can read more on public key encryption here). The private key stays on the local machine and must be kept secure. The public key needs to be copied to the remote machine where the backups from the local machine will be stored. You can use Secure Copy for this: -

scp ~/rsync-key.pub [user]@[remotehost]:~

Finally, you need to put the public key into the “authorized_keys” file on the remote host. SSH into the remote machine using:-

ssh [user]@[remotehost]

and then run the following commands: -


mkdir ~/.ssh
chmod 700 ~/.ssh
mv ~/rsync-key.pub ~/.ssh/
cd ~/.ssh/
touch authorized_keys
chmod 600 authorized_keys
cat rsync-key.pub >> authorized_keys


This creates a hidden .ssh directory with required permissions and then creates the standard authorized_keys file and appends the public key of the local machine to that file. Exit out of the remote machine back into the local machine. You should now be able to ssh into the remote machine from the local without supplying a password. You can test this by running: -

ssh -i ~/rsync-key [user]@[remotehost]

You should now be able to run rsync against a local directory and have that directory backed up on the remote machine all in one command without supplying a password: -

rsync -avz -e "ssh -i ~/rsync-key" /some/directory/path/ [user]@[remotehost]:/backup/directory/path/

You could then add this command to a shell script and execute it as a cronjob on a daily basis, thus ensuring that the remote server has a daily backup of the local machine. The advantage of rsync in terms of network traffic is that it only transfers the data that has changed rather than copying over everything each time, making it quite fast. If you’re doing a full system backup, the “–numeric-ids” switch should be used also: -

rsync --numeric-ids -avz -e "ssh -i ~/rsync-key" /some/directory/path/ [user]@[remotehost]:/backup/directory/path/

If you’re root on the local machine, rsync will do its best to keep the original owner and group of each file/directory on the local “repository”. For that, it will use the remote machine’s /etc/passwd file, meaning that even though the file owner/group will remain constant, the file’s UID (user ID) and GID (group ID) will match those on the remote machine. This can become a problem when you need to restore a full backup, or when accounts are changed on the remote machine.

Using the “–numeric-ids” fixes this problem, as rsync will blindly set the files on the remote to the UID/GID found on the local. This is a must for full backups.

UPDATE 17/09/2011:

This is something I’ve just come across. If you’ve followed this tutorial to the letter and by using your private key to login to the remote host with the command: -

ssh -i ~/rsync-key [user]@[remotehost]

…and your remote host is still asking for a password, check your /var/log/secure log (or wherever your SSHd daemon logs it’s events. If you see the phrase: -

Authentication refused: bad ownership or modes for directory /home/[user]

this is because SSH apparently doesn’t like group write access to that user’s /home directory on the remote host. If you take that off by running the command: -

chmod g-w /home/[user]

…and then try logging into the remote host with SSH using your private key, it should now work. Another thing you can try is setting “StrictMode No” in your /etc/ssh/sshd_config file, this disables Strict modes on directories, which might be a little safer.
This took me two hours of research to figure out, so I’m saving you the trouble! :-)

UPDATE 20/03/2012:

The only other reason that I’ve found where rsync refuses to connect via the public key is because SELinux is enabled. You can temporarily disable it to see if this is the problem with:-

echo 0 >/selinux/enforce

This will turn it off. Use:-

echo 1 >/selinux/enforce

to turn it back on again. If you want to disable SELinux for good, you can edit the following file under RedHat-based systems:-

vi /etc/selinux/config

and change the SELINUX value from “enforcing” to disabled.

SELINUX=disabled

This is not the recommended way to use SELinux (i.e.you should probably use it) but I’ve not looked very deeply into SELinux and for me, it causes more problems than it solves. However I might get around to learning it at some point. If you do know about SELinux, it’s doubtful you needed this tutorial in the first place ;-)