Create secure remote backups using Duplicity – Tutorial
Remote backup is an increasingly popular way to protect your files, and Duplicity makes it easy to implement a secure yet flexible backup system
Duplicity is an easy-to-use system that allows you to make encrypted backups that are stored at a remote location or even in a locally accessible directory. I t has a good selection of networking back-ends (SFTP, SSH, Google storage, S3 etc), so you should be able to fit it into your organisation. The backups are incremental, which saves on bandwidth and storage space when making subsequent backups. Incremental backups also allow the user to step back to a specific point in time to retrieve an older version of a file.
We’ve tried to make the example commands as generalised as possible, so wherever we have put something within square brackets, remove the brackets and insert your own information. Note that Duplicity uses relative paths. So, for example, if you backed up your /etc/directory to a separate archive, you would specify simply ‘resolve.conf’ rather than ‘/etc/resolve.conf’ when retrieving that file. Duplicity is scalable, so it’s possible to carry out a quick backup with a single command or to build something much more elaborate…
Step by Step
Install Duplicity itself along with NcFTP using the package manager for your system (‘sudo apt-get install duplicity ncftp’ on Ubuntu). Don’t forget to compare the version on the website with the version in the repo.
Prepare FTP server
In this tutorial, we’re going to begin with an FTP server as the storage medium. Later on we’ll cover SSH. To test things out, set up an FTP server on a locally accessible machine by installing vsftp with
sudo apt-get install vsftpd
Configure it by editing /etc/vsftpd.conf. Uncomment the lines “local_enable=YES” and “write_enable=YES”. This allows a user on the host system to log in using their normal username and password and to operate on files within their home directory. Restart vsftp by typing
sudo /etc/init.d/vsftpd restart
We’ll do a quick backup to test the setup. Use
mkdir [backups directory]
to create a directory to store backups. Use the following command:
duplicity [source language="directory"][/source] ftp://[username]@[IP address of server]/[backups directory]
Choose a smallish directory to begin with. You’ll then be prompted for a user password for that machine. Following this, enter a GnuPG passphrase. As this is a test, come up with something quick and easy to remember – we’ll create a stronger password later. The backup to the FTP server will now commence.
If everything went okay, Duplicity should have backed up the files and will report this in its output. In addition, the destination directory on the FTP server should now hold three or more encrypted files. These are: the signatures file, the
manifest file and at least one volume file.
If you run exactly the same command again, Duplicity will resynchronise the backup, depositing three or more new files onto the FTP server. In a typical home directory, there may only be a few small changes, for example. The output of Duplicity will reflect the extent of the changes between backups.
You can verify a Duplicity backup with the verify command as so:
duplicity verify ftp://[username]@[address of FTP server]/[backupfolder] [folder that was backed up]
This will alert you to any problems and show you any current inconsistencies between the backups and source directory.
You don’t have to back up to a remote server. Duplicity allows you to back up to a locally accessible directory such as an external hard drive. The command is:
duplicity [source language="directory"][/source] file://[destination directory]
The other options work in exactly the same as for FTP backup.
Single file restore
Let’s try retrieving a single file. Let us say that we have corrupted /etc/network/interfaces and want to retrieve a known good file.
duplicity --file-to-restore /etc/network/interfaces ftp://[username]@[FTP server]/[backups directory] ./interfaces
This command places the file into the current directory.
You can list all of the files in an archive using the following command sequence
duplicity list-current-files ftp://[username]@[FTP server]/[directory]
Using SSH Instead of FTP
Begin by testing that you can SSH into the server from the command line. On the machine that runs Duplicity, run
sudo apt-get install python-paramiko
To back up, use the command
duplicity [source language="directory"][/source] scp://[user]@[SSH server]/[backup directory]
If you use a password to SSH, use the ‘–ssh-askpass’ switch.
You don’t have to use secure keys, if you are confident about the security of your backup medium. However, you can do so, using
. Accept the defaults as you go along, but make a note of the GPG passphrase that you choose when prompted. When it finally finishes, type
and make a note of the ID (eight numbers and letters next to the ‘pub’ entry) of the public key. It’s usual to create separate keys for encryption and the signing of archives, so repeat the procedure to create a second key.
There is a GUI front-end for Duplicity called Déjà Dup. It’s worth considering for simple jobs and for clients who need a bit of control but can’t handle the command line. Fortunately, the actual archives that it creates can be operated on by the regular Duplicity tools.
Recover file by date
You can recover a fi le or directory from a specific time. For example, if you had backed up your /etc directory to its own directory and you want to recover the version of resolv.conf that was known to be working five days ago, then do a single file restore but include the parameter ‘-t 5D’.
From now on, we will use scripts to control Duplicity. Create a file called dupbackup.sh and place the following lines in it:
export FTP_PASSWORD=[ftp password] export PASSPHRASE=[GPG passphrase] duplicity --encrypt-key "[encrypt key]" --sign-key "[gpg ID]" [sign key] ftp://[username]@[IP address of server]/[backups directory]
The retrieval script (call it dupretrieval.sh) is similar, but the command line becomes:
duplicity --encrypt-key "[encrypt key]" --sign-key "[sign key]" ftp://[username]@[IP address of server]/[backups directory]
Remember to chmod+x them both before use.
Excluding files (size, pattern)
Exclude files from the backup by using a list file specified with the ‘–exclude-filelist’ switch. Put a minus sign in front of files and directories to exclude and the list file can include wildcards. To exclude files over a certain size, use the ‘find’ command – for example:
find . -size +100M -print > oversize.txt
You can add a crontab to run your backup script at regular intervals. For example, type
sudo crontab -e
and then add a line such as
0 2 * * 1 /home/myuser/mybackup.sh
to run a backup at 2am once each day.
Full system backup
Modify the existing backup script, save it under a new name and make it executable. The command sequence is now
duplicity --encrypt-key "[encrypt key]" --sign-key "[sign key] --exclude-filelist=exclusion_list/ ftp://[FTP user]@[FTP server]/[backup folder]
This starts backing up files from the root folder, and must be run as root. You also need to create an ‘exclusion_list’ file with the following entries on separate lines: – /sys – /dev – /proc – /tmp – /mnt (dashes included).
There is more than one approach to recovering an entire system, but the easiest is to reinstall the system as usual and then replace the files with those from the backup. Exercise extreme caution when carrying this out. Unfortunately, Duplicity won’t overwrite existing files, so you have to do this manually. Create a new script with the command sequence:
duplicity --encrypt-key "encrypt key" --sign-key "sign key" ftp://[FTP server]@[FTP server]/[backup directory] /[destination directory]
Now wipe the existing fi les from the destination disk and use
rsync -avz [source language="dir"][/source] [destination drive root]
to copy the backup files across.