A FreeNAS box (okay, TrueNAS Core as we should now call it) is an ideal place to perform set-and-forget, reliable backups of my Linux laptop, given that both devices now use the OpenZFS 2.x filesystem. This howto is not intended to be a ZFS primer – I am assuming the reader already understands ZFS terminology – so suffice it to say we will take scheduled snapshots of the entire current laptop disk state, then copy those snapshots to a location on the TrueNAS filesystem, available in a format ready for off-site backup using Duplicacy. For reasons of security and isolation I want to use a FreeBSD 12 jail as the target, rather than the FreeNAS operating system itself.

This howto explains how I achieved this aim, leveraging zfs_autobackup and FreeBSD’s excellent periodic utility. zfs_autobackup¬† uses ZFS’ snapshotting and copying features to perform backup and archiving from one machine to another over ssh, optimized for speed, and also handles aging-out older snapshots over time. periodic is a feature-complete task scheduling tool. Note that I will use the names TrueNAS and FreeNAS interchangeably in this article, since I have written it on the cusp of the re-branding of FreeNAS to TrueNAS Core.

My starting point is the assumption that you’ve already created a standard FreeNAS jail that is going to perform your backups. In my case, this jail also handles offsite backups to Backblaze B2 using the wonderfully robust duplicacy tool. This is relevant only because my approach is that anything under /mnt is in scope for offsite backup, hence I desire that my laptop datasets appear mounted somewhere under /mnt once the backup has completed.

    1. Upgrade ZFS in the jail
      N.B. This step will become unnecessary once TrueNAS Core ships with FreeBSD 13.x.
      At time of writing, whilst TrueNAS Core 12 supports OpenZFS 2.0, the FreeBSD version on which it is based (12.2) does not. This means that jails do not include the OpenZFS 2.0 userland tools. To avoid weird version incompatibility errors it is strongly advised to copy the zfs binary (and dependent libraries) from the base TrueNAS OS into the jail. This situation is well explained by this TrueNAS support ticket, which also includes a handy script to perform the copying.
    2. Install zfs-autobackup
      First of all, install Python into the jail (pkg install python37), then follow the zfs-autobackup installation howto. This will guide you to install the software on the backup jail, set up the jail’s ability to ssh to the machine to be backed up, and apply the necessary custom zfs property to the datasets on the machine to be backed up. I would also highly recommend following the Performance Tips as part of installation, since they can make a noticeable difference to how long backups take. Remember, the first one will theoretically be the slowest one of all, so best to get it tuned from the start!
    3. Create destination dataset
      zfs-autobackup copies zfs datasets to zfs datasets, so create a zfs dataset on the FreeNAS host system pool for the backups to go to. Since this dataset is being used explicitly to store backups of snapshots, it might make sense during creation of the dataset set the ‘Snapshot directory’ property to Visible. There is also the option to set the dataset read-only to avoid accidental alterations that could casuse zfs-autobackup to throw ‘target altered’ errors on subsequent backups. Read-only datasets can still zfs recv data (which is what zfs-autobackup leverages) and alterations to dataset properties (zfs get/set) are allowed, but at the filesystem level no other process can alter its content.
    4. Assign dataset to jail
      So far so good, but by default jails cannot ‘see’ the FreeNAS system datasets – try a ‘zfs list’ to confirm – so we need to leverage a FreeNAS ZFS feature to change that. With the jail shut down, go into Edit.. Jail Properties, and tick allow_mount and allow_mount_zfs to allow root to mount and unmount zfs datasets the jail has assigned to it. Next we need to assign the destination dataset to the jail. In the jail’s Edit.. Custom Properties, tick jail_zfs and in the jail_zfs_dataset field enter the dataset name _without_ the pool name prepended. In the jail_zfs_mountpoint field enter the dataset name prepended by a / (the folder in which mounts are mounted – /mnt by default – is implicit and should not be included here). Now save the jail settings and start it. Keep in mind that once assigned to a jail, the dataset cannot be mounted by the FreeNAS host. It effectively now ‘belongs’ to this jail alone. You can immediately zfs list from within the jail and find it now available.
    5. Run the initial full backup
      The following command should now be run from within the jail to perform the initial backup of your source machine to the dataset mounted to the jail. Remember that some of the performance tips listed in the zfs_autobackup installation article are not part of the backup command, so perhaps review those again at this point. The following command took ~12 hours to back up ~550GB over my 1Gbps LAN, achieving an average speed of 10MB/s:

      zfs-autobackup –ssh-source carbon.local remote med/carbon-backup –progress –verbose –clear-mountpoint –clear-refreservation –zfs-compressed –no-holds

      Notes:

      • –clear-mountpoint disables automatic mounting for the datasets being backed up to the jail, allowing them to be mounted in a more logical layout for a backup under /mnt on the target (see next step of this howto)
      • –clear-refreservation prevents the back-up datasets reserving extra space for growth on the target system.
      • –zfs-compressed copies compressed datasets as-is and will speed up the transfer. No good if the target dataset uses a different compression algorithm to the source.
      • –no-holds does not enable the ‘hold’ flag on the target snapshots. Apparently this speeds up transfer, but be aware that it will allow you to delete the target datasets without the enforced ‘Are you sure?’ of having to run a zfs command to release the held snapshots first.
      • –progress –verbose outputs verbose progress info. –no-progress disables output of percentage progress and time remaining, which is preferable when run in an automated manner from a script. Note that –progress is the cause of a performance issue introduced in ZFS 2.0.1 and not yet resolved at time of writing.
      • there is a –debug switch that provides very helpful (but not excessive) output if you run into any errors.
    6. Modify mount points on destination
      The backed-up datasets bring with them their mountpoints relative to the source filesystem, which will be relative to /mnt on the backup jail. If your source machine runs multiple datasets mounted in a filesystem hierarchy. such as /, boot, root and home, for example, these would end up mounted directly into /mnt. For neatness and clarity it would be preferable for them to appear in a subdirectory under /mnt, named for the source machine, hence the reason for disabling their automated mounting during the backup using –clear-mountpoint. With the initial backup complete, we can use zfs set mountpoint to one-time modify the mount points for all the destination datasets, then mount them. It is important to note that the mount points must not be below the mountpoint of the ‘parent’ dataset on the destination, nor can they be sub-directories of one another, else next time you run a backup you will get ‘destination has been modified since most recent snapshot’ errors. Here then is the script I used to modify the mount points and mount the backed-up datasets:

      zfs set mountpoint=/carbon-mounts/boot med/carbon-backup/boot
      zfs set mountpoint=/carbon-mounts/rpool med/carbon-backup/rpool
      zfs set mountpoint=/carbon-mounts/home med/carbon-backup/rpool/HOME
      zfs set mountpoint=/carbon-mounts/root med/carbon-backup/rpool/ROOT
      zfs set mountpoint=/carbon-mounts/gentoo-root med/carbon-backup/rpool/ROOT/gentoo
      zfs mount med/carbon-backup/boot
      zfs mount med/carbon-backup/rpool
      zfs mount med/carbon-backup/rpool/HOME
      zfs mount med/carbon-backup/rpool/ROOT
      zfs mount med/carbon-backup/rpool/ROOT/gentoo

      The destination is now ready for subsequent incremental backups, and will retain these new mount points across backups.

    7. Set up scheduled backups
      The backup is initiated from the server side, meaning we can leverage FreeBSD’s excellent periodic utility to set up a daily execution of zfs_autobackup. Setting up a schedule is as simple as adding a shell script in /etc/periodic/daily that runs the same zfs_autobackup command as used for the initial backup, but adding the –no-progress option since we’re not watching the output.
    8. Work around reading-snapshots-in-a-jail bug
      For reasons as yet unexplained, attempts to ls the content of .zfs/snapshot on datasets from within a jail return ‘Operation not permitted’. This is a problem for me since I assume my file-based off-site backup tooling will run into this bug. I don’t know whose it is (OpenZFS, FreeBSD or TrueNAS), but fortunately someone discovered a simple workaround – list the directory contents from the base TrueNAS shell and thereafter it’ll work from within the jail. I’ve set up a dead simple cron job in TrueNAS Core that runs every hour to ls the content of all the jailed datasets.