September 8, 2015

Some Shell Scripts for Duplicity

Some days ago I accidentally noticed that my ISP upped the meager free online storage capacity included in my contract to a whopping 1TB. This is finally a size where one can think about backing up data to ‘the cloud’.

I updated my old duplicity scripts that I did not use for some time and thought I can put them to this place; maybe they are useful to somebody. The files are also available in my github. Download them from there. Here's only a brief description of some of the scripts and some background info about duplicity and things.

As duplicity uses GnuPG to encrypt data, you should create yourself a new GPG key dedicated to this purpose. Add this key to the key ring of a user of your server or PC. Also mount your remote storage to some place, e.g., using davfs or sshfs.

config.conf

For various settings, I created a configuration file that is later included into the scripts:

PASS='pass'
KEY='keyid'
EXCL='exclude.list'
SRC=('/path/one' '/path/two' .. 'path/n')
DEST='file:///path/to/webdav'
REST='/path/to/restore/dir'

The variables do the following:

  • PASS is the gpg key's password.
  • KEY is the gpg key's identifier without “0x”.
  • EXCL is the path of the exclude file; here you can, for instance, select not to backup *.bak files or so.
  • SRC a list of those paths that need backing up, i.e., data sources.
  • DEST is the mountpoint of the store, i.e., the backup destination.
  • REST is the path the restore script will restore all data to in case of a harddrive meltdown or so.

Duplicity and the GnuPG key's password

Duplicity has one problem when you want to use it in a script. It needs the gpg key's password. This is especially annoying if you want to run the script as a cronjob. But the duplicity guys provide a solution for that:

...
export PASSPHRASE=$PASS
export SIGN_PASSPHRASE=$PASS
duplicity manyParameters --encrypt-key $KEY --sign-key $KEY evenMoreParameters
unset PASSPHRASE
unset SIGN_PASSPHRASE

Basically, you export the password as an environmental variable, duplicity reads this variable and starts backing up data. As soon as duplicity terminates, you delete (unset) the password. Not really nice, but it gets the job done.

The backup script

...
for LOCAL in ${SRC[*]};
do
    REMOTE=$DEST/${LOCAL##*/}
    echo Backing up $LOCAL to $REMOTE
    duplicity -v4 --encrypt-key $KEY --sign-key $KEY --exclude-globbing-filelist $MYPATH/$EXCL $LOCAL $REMOTE
done
...

The script itself is not very complicated. For each of the paths specified in $SRC, it calls duplicty and backs up data to the path specified by $REMOTE. This is simply a subdirectory in $DEST named like the last bit of the current $SRC.

The Rest

The remaining scrips work pretty much as the backup script. list.sh creates a list of all files included in the backup. That is quite handy to check if the backup works and it is also useful to find a missing file in the backup that can be individually restored. restoreAll.sh does exaclty what the name says. It restores everything to $REST.

© holger 2015 - 2020 |