Official website for Linux User & Developer
FOLLOW US ON:
May
28

Create secure remote backups using Duplicity – Tutorial

by Michael Reed

Remote backup is an increasingly popular way to protect your files, and Duplicity makes it easy to implement a secure yet flexible backup system

Duplicity is an easy-to-use system that allows you to make encrypted backups that are stored at a remote location or even in a locally accessible directory. I t has a good selection of networking back-ends (SFTP, SSH, Google storage, S3 etc), so you should be able to fit it into your organisation. The backups are incremental, which saves on bandwidth and storage space when making subsequent backups. Incremental backups also allow the user to step back to a specific point in time to retrieve an older version of a file.

We’ve tried to make the example commands as generalised as possible, so wherever we have put something within square brackets, remove the brackets and insert your own information. Note that Duplicity uses relative paths. So, for example, if you backed up your /etc/directory to a separate archive, you would specify simply ‘resolve.conf’ rather than ‘/etc/resolve.conf’ when retrieving that file. Duplicity is scalable, so it’s possible to carry out a quick backup with a single command or to build something much more elaborate…

Dropbox
Duplicity can store your encrypted backups on an FTP server, and it has a host of other back-ends such as local file access or SSH

Resources

Duplicity
vsftpd
NcFTP

Step by Step

Step 01

Installation

Install Duplicity itself along with NcFTP using the package manager for your system (‘sudo apt-get install duplicity ncftp’ on Ubuntu). Don’t forget to compare the version on the website with the version in the repo.

Step 02

Prepare FTP server

In this tutorial, we’re going to begin with an FTP server as the storage medium. Later on we’ll cover SSH. To test things out, set up an FTP server on a locally accessible machine by installing vsftp with

sudo apt-get install vsftpd

Configure it by editing /etc/vsftpd.conf. Uncomment the lines “local_enable=YES” and “write_enable=YES”. This allows a user on the host system to log in using their normal username and password and to operate on files within their home directory. Restart vsftp by typing

sudo /etc/init.d/vsftpd restart

Step 03

Simple backup

We’ll do a quick backup to test the setup. Use

mkdir [backups directory]

to create a directory to store backups. Use the following command:

duplicity  ftp://[username]@[IP address of server]/[backups directory]

Choose a smallish directory to begin with. You’ll then be prompted for a user password for that machine. Following this, enter a GnuPG passphrase. As this is a test, come up with something quick and easy to remember – we’ll create a stronger password later. The backup to the FTP server will now commence.

Step 04

Examine output

If everything went okay, Duplicity should have backed up the files and will report this in its output. In addition, the destination directory on the FTP server should now hold three or more encrypted files. These are: the signatures file, the
manifest file and at least one volume file.

Step 05

Repeat test

If you run exactly the same command again, Duplicity will resynchronise the backup, depositing three or more new files onto the FTP server. In a typical home directory, there may only be a few small changes, for example. The output of Duplicity will reflect the extent of the changes between backups.

Step 06

Verify backup

You can verify a Duplicity backup with the verify command as so:

duplicity verify ftp://[username]@[address of FTP server]/[backupfolder] [folder that was backed up]

This will alert you to any problems and show you any current inconsistencies between the backups and source directory.

Step 07

Local backup

You don’t have to back up to a remote server. Duplicity allows you to back up to a locally accessible directory such as an external hard drive. The command is:

duplicity  file://[destination directory]

The other options work in exactly the same as for FTP backup.

Step 08

Single file restore

Let’s try retrieving a single file. Let us say that we have corrupted /etc/network/interfaces and want to retrieve a known good file.

duplicity --file-to-restore /etc/network/interfaces ftp://[username]@[FTP server]/[backups directory] ./interfaces

This command places the file into the current directory.

Step 09

List files

You can list all of the files in an archive using the following command sequence

duplicity list-current-files ftp://[username]@[FTP server]/[directory]

Step 10

Using SSH Instead of FTP

Begin by testing that you can SSH into the server from the command line. On the machine that runs Duplicity, run

sudo apt-get install python-paramiko

.

To back up, use the command

duplicity  scp://[user]@[SSH server]/[backup directory]

If you use a password to SSH, use the ‘–ssh-askpass’ switch.

Step 11

Generate key

You don’t have to use secure keys, if you are confident about the security of your backup medium. However, you can do so, using

gpg --gen-key

. Accept the defaults as you go along, but make a note of the GPG passphrase that you choose when prompted. When it finally finishes, type

gpg --list-keys

and make a note of the ID (eight numbers and letters next to the ‘pub’ entry) of the public key. It’s usual to create separate keys for encryption and the signing of archives, so repeat the procedure to create a second key.

Step 12

Déjà Dup

There is a GUI front-end for Duplicity called Déjà Dup. It’s worth considering for simple jobs and for clients who need a bit of control but can’t handle the command line. Fortunately, the actual archives that it creates can be operated on by the regular Duplicity tools.

Step 13

Recover file by date

You can recover a fi le or directory from a specific time. For example, if you had backed up your /etc directory to its own directory and you want to recover the version of resolv.conf that was known to be working five days ago, then do a single file restore but include the parameter ‘-t 5D’.

Step 14

Create scripts

From now on, we will use scripts to control Duplicity. Create a file called dupbackup.sh and place the following lines in it:

export FTP_PASSWORD=[ftp password]
export PASSPHRASE=[GPG passphrase]
duplicity --encrypt-key "[encrypt key]" --sign-key "[gpg ID]" [sign key] ftp://[username]@[IP address of server]/[backups directory]

The retrieval script (call it dupretrieval.sh) is similar, but the command line becomes:

duplicity --encrypt-key "[encrypt key]" --sign-key "[sign key]" ftp://[username]@[IP address of server]/[backups directory]

Remember to chmod+x them both before use.

Step 15

Excluding files (size, pattern)

Exclude files from the backup by using a list file specified with the ‘–exclude-filelist’ switch. Put a minus sign in front of files and directories to exclude and the list file can include wildcards. To exclude files over a certain size, use the ‘find’ command – for example:

find . -size +100M -print > oversize.txt

Step 16

Automating backups

You can add a crontab to run your backup script at regular intervals. For example, type

sudo crontab -e

and then add a line such as

0 2 * * 1 /home/myuser/mybackup.sh

to run a backup at 2am once each day.

Step 17

Full system backup

Modify the existing backup script, save it under a new name and make it executable. The command sequence is now

duplicity --encrypt-key "[encrypt key]" --sign-key "[sign key] --exclude-filelist=exclusion_list/ ftp://[FTP user]@[FTP server]/[backup folder]

This starts backing up files from the root folder, and must be run as root. You also need to create an ‘exclusion_list’ file with the following entries on separate lines: – /sys – /dev – /proc – /tmp – /mnt (dashes included).

Step 18

System restore

There is more than one approach to recovering an entire system, but the easiest is to reinstall the system as usual and then replace the files with those from the backup. Exercise extreme caution when carrying this out. Unfortunately, Duplicity won’t overwrite existing files, so you have to do this manually. Create a new script with the command sequence:

duplicity --encrypt-key "encrypt key" --sign-key "sign key" ftp://[FTP server]@[FTP server]/[backup directory] /[destination  directory]

Now wipe the existing fi les from the destination disk and use

rsync -avz  [destination drive root]

to copy the backup files across.

Tags: ,
  • Tell a Friend
  • Follow our Twitter to find out about all the latest Linux news, reviews, previews, interviews, features and a whole more.