Here’s the process to set up duplicacy to send backup data to a remote SFTP server, with unattended operation supported through using public key authentication. I will assume you already have:

  • an account set up on the destination server
  • generated an ssh keypair with ssh-keygen and NOT specified a passphrase (or add the passphrase in the preferences file for unattended backup, using duplicacy set -key ssh_passphrase -value <passphrase>)
  • the public key has been copied into the ~/.ssh/authorized_keys file on the server
  • the private key file is in ~/.ssh on your source system

If you already have duplicacy set up and backing up to an alternate location, you can retain but disable that config by renaming the .duplicacy subfolder of wherever duplicacy is executed from. Yes, one can support multiple configurations within one preferences file by using the -storage-name init parameter for each setup, but I prefer entirely separating the configs, though I still use -storage-name to remind myself what each one is for.

  1. First of all, change into the root folder of your source data to be backed up, then initialise duplicacy with the following command. Note that -encrypt only encrypts the config file stored server-side. If you wish to fully encrypt the backup, you’ll need to create a second keypair that must use the RSA algorithm, ensure the public key is in PEM format and add -key <path/to/public-key-filename.pem> to the command: 
    duplicacy init -encrypt -storage-name <name-for-your-SFTP-destination> <name-for-your-source-data> sftp://<username>@<server-domain-name-or-IP-address>:<port-number-SSH-is-listening-on>/<subfolder-in-which-to-store-the-backup>
  2. You’ll be prompted for the encryption password.
  3. Store the encryption password in your preferences file so that it can run unattended when called by cron/periodic in future: 
    duplicacy set -key password -value <your-encryption-password>
  4. Store the path to your private authentication key in your preferences file so that duplicacy knows which key to use for authentication: 
    duplicacy set -key ssh_key_file -value <path/to/private-key-file>
  5. Since your .duplicacy/preferences file now stores secrets, make sure it is only read-write to whichever user account will run duplicacy during unattended backups. By default this would be root.
  6. Test your setup by running a dry-run backup. Although this won’t copy any files, if you have a large data set (hundreds of thousands of files) be aware that the first phase of the process, indexing, may well take over an hour alone. As a rough idea, my system with ~400,000 files for ~4TB takes 1.5 hours to index: 
    duplicacy backup -dry-run -stats
  7. If your test run all goes well you can run your initial backup. Be aware that for a large dataset (>1TB) over the average internet connection this will take multiple days! You can speed up the file copy portion of the process by using -threads which will open multiple connections to the SFTP server, the cost being increased memory usage on the client side. With only four threads my system peaked at around 8GB being used by duplicacy’s process, so be careful to ensure you have the memory headroom, else the process will slow to a crawl. 
    duplicacy backup -threads 4 -stats
  8. To configure unattended backup, follow the appropriate portions of this article to set up periodic and the set of scripts the article provides to invoke your backup config. Obviously enough, ignore the parts about duplicacy initialization to set up BackBlaze as destination!
  9. I recommend adding >> backup.log 2>&1 to your scripted duplicacy invocation so as to log its output to a file. That way you can periodically review the log to see what data is being copied and how long each backup is taking, such that you can tweak the number of threads up if you think it could afford to perform copying more quickly, or down if it seems to be browning out on memory.