rdiff-backup

Backing Up Data with rdiff-backup

rdiff-backup is an excellent backup program that creates incremental backups onto a disk drive. It can operate over the network and is very efficient (it uses the rsync library, in fact). By default, it performs its operations through SSH, which is something I like.

Just like with our rsync examples in the previous section, the most recent backup is a complete mirror of the source files. The previous versions of the files, however, are maintained by storing reverse diffs of any modified files. Both text and binary files are stored in this fashion, which makes for very efficient usage of disk space.

You must have the rdiff-backup program installed on all systems involved in the backup process—the server(s) and every client. You can obtain the software and find additional documentation at http://rdiff-backup.stanford.edu.

Creating Incremental Backups with rdiff-backup

Let's start with a basic script that performs backups of specified directories onto a local backup directory:

#!/bin/bash
DIRS="etc home var usr/local"
DEST="/mnt/backup"
cd /
for dir in $DIRS ; do
rdiff-backup $dir "$DEST/$(basename $dir)"
done

You can run this script as often as you like. I would recommend running it once per day. If the data is very important (like a corporate CVS repository), running it once per hour wouldn't be unreasonable.

Modifying this script to back up to a remote system is as simple as changing the $DEST variable. Here is an example:

DEST="backup.mydomain.com::/mnt/backup"

This causes rdiff-backup to connect to the host backup.mydomain.com via SSH and place the backup in its directory /mnt/backup. Regardless of what system contains the backup data, it will be almost an exact mirror of the source directory. Permissions, special files, hard/soft links, and ownership will all be preserved, if possible.

The backup directory is exactly like the original, except for the additional rdiff-backup-data/ directory. This directory contains the backup logs and, most importantly, the diff files that can be used to generate older versions of files. It is very important that you do not manually change the files in the backup directory in any way. Doing so can permanently damage the backup data, probably preventing the correct recovery of any versions of the modified files.

If you are backing up the data to a remote host, you will usually want to use a directory on that system based on your system's hostname. We have done this in previous discussions of backup tools. Here is a simple example:

#!/bin/bash

DIRS="etc home var usr/local"
DEST="/mnt/backup"
shorthost='hostname sed 's/\..*$//''
cd /
for dir in $DIRS ; do
rdiff-backup $dir "$DEST/$shorthost/$(basename $dir)"
done

Removing Old Backup Data

If left alone, the backup directory perpetually increases in size, unless, of course, the source data is static. For this reason, you will want to clean out the old backup data every week or so. The following script takes care of this for you:

#!/bin/bash
DEST="/mnt/backup"
rdiff-backup --remove-older-than 1M $DEST

The string 1M is a time specification representing one month. This means that all data relating to files or versions of files more than one month old will be deleted.

Time specifications can be given in other formats as well. You can use the standard "seconds since the epoch," or you can use a string such as 2003-01-06T23:30:45-04:00 (which provides the date, the time, and the local time zone's deviation from Coordinated Universal Time (UTC)). Simpler date-only formats, such as 2003-01-07 and 12/31/2002 are also supported.

The most useful specifications, however, are the relative ones. The letters s, m, h, D, W, M, and Y represent seconds, minutes, hours, days, weeks, months, and years, respectively. You can combine these as necessary; for instance, 1M2D3h specifies one month, two days, and three hours ago.

Sometimes it is nice to be able to remove data only when the backup drive is filling up. Here is a script that is fairly cryptic, not too efficient, but very flexible. You can specify a directory to monitor and a maximum amount of space to be used on that drive. If the drive is too full, this script searches for all backups under that directory created by rdiff-backup. When you run the following script, rdiff-backup removes the oldest increment and repeats the whole process, if necessary. You should run this script regularly from your backup server's cron daemon:

#!/bin/bash
# Drive or directory to clean up
prune="/mnt/backup"
# The maximum used space allowed on the drive (percent)
cutoff="80"
# Removes one increment from each rdiff-backup repository until the
# drive falls below the specified cutoff
until df "$prune" awk -v "cutoff=$cutoff" '/^\// {

gsub(/%$/, "", $5);
if ($5 > cutoff)
exit 1
}' ; do
# Too full, clean out some old data
# First, find and store rdiff-backup repositories
[ -z "$DIRS" ] && {
for dir in 'find $prune -name rdiff-backup-data -type d' ; do
DIRS="$DIRS $(echo "$dir" sed 's/\/rdiff-backup-data//')"
done
}
# Now, delete the oldest increment
for dir in $DIRS ; do
del='rdiff-backup -l "$dir" sed -n \
's/^ *increments\.\(....-..-.....:..:..-..:..\)\.dir.*/\1/p' \
head -n2 tail -n1'
echo >&2
echo "Cleaning $dir..." >&2
rdiff-backup --remove-older-than "$del" "$dir" >&2 {
echo "Only one increment left for $dir (skipping)" >&2
DIRS='echo "$DIRS" sed "s= *$dir *=="'
}
done
# Prevent an infinite loop from occurring
[ -z "$DIRS" ] && {
echo >&2
echo "No rdiff-backup data directories left to prune!" >&2
exit 1
}
done
exit 0

This script assumes that your drive is sufficiently large to keep several increments in each repository. At the very least, it keeps all of the current data and the most recent increment. This script is careful to exit if no more data can be removed; otherwise an infinite loop would occur.

Restoring Data

Once you have a backup directory, you can quickly display a listing of the increments available. An increment is some previous date for which exact files are available. Whenever you run rdiff-backup, the current mirror is updated and a new increment is created, still allowing you to retrieve the previous set of data.

Here is an example listing of increments:

% rdiff-backup -l /some/backup/directory
Found 4 increments:
increments.2003-01-02T23:30:45-04:00.dir Thu Jan 2 23:30:45 2003

increments.2003-01-03T23:31:40-04:00.dir Fri Jan 3 23:31:40 2003
increments.2003-01-04T23:31:54-04:00.dir Sat Jan 4 23:31:54 2003
increments.2003-01-05T23:32:02-04:00.dir Sun Jan 5 23:32:02 2003
Current mirror: Mon Jan 6 23:32:18 2003

There are two ways to restore data from an rdiff-backup archive. One is to look through the data and run rdiff-backup on one of the increment files. I'll let you read the rdiff-backup man page if you want to try that one. The second method, which is my favorite, is to use the –restore-as-of(-r) switch.

Let's assume that the backup archive shown in the previous increment listing was created by the following backup script, executed at around 23:30 every day:

rdiff-backup /etc /some/backup/directory

Now, assuming it is Tuesday at 16:00, and I want to restore the /etc/ directory as it was yesterday at this time, I would run the following command:

rdiff-backup -r 1D /some/backup/directory /tmp/etc.old

Since no backup was performed exactly one day ago, the next oldest increment is used. In this case, that would be the backup performed on Sunday night (2003-01-05T23:32:02-04:00). I could also have specified this full date string instead of 1D as the argument to the -r switch.

The exact state of the /etc/ directory when that backup was performed is now re-created in the local directory /tmp/etc.old/. You can then do anything you would like within this temporary directory.

Excluding Files When Using rdiff-backup

A wide variety of command-line options allows you to control which files are included in the backup process. Here are the ones more commonly used:

–exclude pattern: Files matching the pattern should be excluded. Matching directories, and their contents, will be excluded.

–exclude-device-files: Causes all device files to be ignored.

–exclude-filelist filename: All files listed in the file filename are excluded from the backup operation.

–exclude-filelist-stdin: A list of files can be sent in through stdin. These files will be excluded.

–exclude-other-filesystems: Only files on the same physical filesystem as the base source directory are backed up.

–exclude-regexp regex: Files matching the regular expression will be excluded. Python-style regular expressions are used. Files within matching directories are processed separately.

–include pattern: Files matching the pattern should be included. Matching directories, and their contents, will also be included.

–include-filelist filename: Only files listed in the file filename are included in the backup operation.

–include-filelist-stdin: A list of files can be sent in through stdin. Only files found in this list will be part of the backup.

–include-regexp regex: Files matching the regular expression will be included. Python-style regular expressions are used. Files within matching directories are processed separately.

Regardless of the number of these options specified, each rule is applied to each file. The first rule that matches the file determines its fate. If no rule matches, the default action is to include the file. For instance, let's say that you want to back up the /var/ directory, except for the /var/spool/ directory, but you do want /var/spool/mail/. You can do this:

rdiff-backup --include /var/spool/mail --exclude /var/spool /var /backup

A file pattern (used by the –include and –exclude options) can contain the standard shell globbing characters (*, +, and []). Neither * nor + can match a /; this is important since the full pathname of each file is compared. The special sequence **, however, does match any number of characters, including forward slashes. The pattern can begin with the string ignorecase: to—you guessed it—make the pattern case-insensitive. Be sure to quote these patterns so that the shell doesn't interpret them before rdiff-backup gets a chance at them.