My laptop runs ZFS as its root file system (see this blog post) — meaning that I can snapshot my root file system and I can send it to another machine as a backup very easily. Unfortunately, while ZFS provides the raw functionality, there is no great tool to manage offsite backups and retention. To ease this pain, I wrote/forked and packaged a few helper scripts which I called zfsu, a collection of ZFS utilities.
It consists of the following tools: zfsu tx (aka zfstx) maintains a mirror of a ZFS pool over the network. zfsu ret (aka zfsret) is a simple script to apply local retention (destroy snapshots) of a file system and its snapshots. zfsu res (aka zfsres) is a script to resilver a slow mirror, e.g. a HDD disk if mirrored with a SSD.
Content
0. Code available on GitHub
All the code for this post is available in a zfsu Project on GitHub. Feel free to poke around there and/or steal bits and pieces from it.
1. Installation
I host a package of the scripts in my Debian repository, so if you are Debian/Ubuntu based, feel free to install it like this:
|
1 2 3 4 5 6 7 |
# Add my repository wget -qO - http://archive.philippheckel.com/apt/Release.key | sudo apt-key add - sudo sh -c "echo deb http://archive.philippheckel.com/apt/release/ release main > /etc/apt/sources.list.d/archive.philippheckel.com.list" sudo apt-get update # Install zfsu sudo apt-get install zfsu |
If you’re using a different system, simply download and place these scripts in your PATH.
2. zfsu tx (aka zfstx)
zfstx lets you pull ZFS snapshots from a remote host into the local zpool, meaning that you need to install this script on the backup machine itself, not on the machine that’s being backed up. While this may seem strange at first, it’s great if you’re backing up many different machines, because you can manage the schedule and logic in one place.
The tool was originally developed by Jaako as part of his zfs_transfer tool. I merely adjusted it a little and packaged it for my needs. Many thanks to him for this great tool.
2.1. Usage
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
$ zfsu tx Usage: zfstx [OPTIONS] <remote-host>:<remote-fs> <local-fs> Pull ZFS snapshots from a remote host into the local zpool. Arguments: <remote-host> - Remote host, e.g. myhost <remote-fs> - Filesystem on remote host, e.g. tank/home <local-fs> - Filesystem on local host, e.g. backuppool/myhost/home Options: -k, --keep <count> - preserved history length -b, --mbuffer <bufsz> - mbuffer buffer size, default 4G -p, --port <port> - custom SSH port, default 22 -P, --no-pigz - disable pigz -n, --dry-run - Don't apply changes, just print (experimental) |
2.2. Examples
|
1 2 3 4 5 6 7 |
$ zfstx platop:tank/home/pheckel tank/backups/platop/home/pheckel # Pull all (missing) snapshots from host "platop" into the local pool "tank" # and don't apply any retention. $ zfstx --keep 5 platop:tank/home/pheckel/vms tank/backups/platop/home/pheckel/vms # Pull all (missing) snapshots from host "platop" into the local pool "tank" # and only keep the 5 latest snapshots locally. |
3. zfsu ret (aka zfsret)
zfsret is a simple script to apply local retention (destroy snapshots) of a file system and its snapshots. Its meant to complement the zfstx utility to only maintain a certain number of snapshots on the backup host. It can also be used to locally prune snapshots on demand, or regularly.
The retention logic is pretty simple as you can only specify how many of the last snapshots to keep. There is no sophisticated logic (yet) to keep X per week or Y per month.
3.1. Usage
|
1 2 3 4 5 6 7 8 9 10 |
$ zfsu ret Usage: zfsret [OPTIONS] <local-fs> <keep> Destroy local ZFS snapshots for a specific filesystem. Arguments: <local-fs> - Filesystem on local host, e.g. backuppool/myhost/home <keep> - Number of snapshots to keep Options: -n, --dry-run - Don't apply changes, just print |
3.2. Examples
|
1 2 3 |
$ zfsu ret tank/home/pheckel 10 # Destroy all but 10 snapshots of filesystem tank/home/pheckel # This is not recursive (no -r)! |
4. zfsu res (aka zfsres)
zfsres is a tool specifically written for a ZFS mirror/RAID1 between a slow/HDD and a fast/SSD disk. zfsres will regularly resilver the slow HDD and thereby not make the array slow for normal write/read operations, but still allow for a RAID1 configuration of two disks of vastly different speeds.
This tool covers a pretty exotic use case. I wrote it because my laptop has a SSD and a HDD, and I wanted to keep a second copy on the HDD. Note that this tool only makes sense in this scenario.
4.1. Usage
|
1 2 3 4 5 6 7 |
$ zfsu res Usage: zfsres <pool> <slow-mirror> Enable slow mirror(s) and wait for them to be resilvered and exit. Arguments: <pool> - Name of the ZFS pool, e.g. tank <slow-mirror> - Description of a slow mirror device, e.g. wwn-0x50004cf20c41a05b |
4.2. Examples
|
1 2 3 |
$ zfsres tank wwn-0x50004cf20c41a05b # Will online the disk wwn-0x50004cf20c41a05b and wait for resilvering to # complete, and then offline the disk again. |
5. Example: My setup
I use these scripts to back up 5 different machines, some of which have ZFS file systems, and others that do not. My backup hub’s ZFS pool looks like this (shortened):
|
1 2 3 4 5 |
tank/backups/laptop1 1.09T 1.29T 9.65G /tank/backups/laptop1 tank/backups/box 207G 1.29T 9.51G /tank/backups/box tank/backups/box2 1.07G 1.29T 868M /tank/backups/box2 tank/backups/laptop2 893G 1.29T 14.0G /tank/backups/laptop2 tank/backups/blurp 419G 1.29T 6.60G /tank/backups/blurp |
Backups of these machines are tiggered in a cronjob by a custom wrapper script called gobackup, which uses the zfsu utilities. It pretty much looks like this (representative excerpt):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
#!/bin/bash # Example 'laptop1': a system that does not use ZFS as a file system # We just rsync from there and manage snapshots and retention locally laptop1_root() { ( flock -n -x 9 || { echo "Already running."; return; } rsync \ -e "ssh -o ConnectTimeout=15" \ -av \ --delete \ --delete-excluded \ --one-file-system \ --compress \ root@laptop1:/ \ /tank/backups/laptop1/ \ && zfs snapshot tank/backups/laptop1@`date +%y%m%d%H%M` \ && zfsret tank/backups/laptop1 100 ) 9> /var/lock/gobackup_laptop1_root } # Example 'laptop2': a system that does use ZFS. A single command # pulls the latest snapshots to the backup hub. laptop2_home() { zfstx \ --mbuffer 1G \ --keep 9000 \ laptop2:tank/home/pheckel \ tank/backups/laptop2/home/pheckel } laptop1_root laptop2_home |
Comments are closed.
Recent Comments