Duplicati

Always have a backup plan¹

duplicati Screenshot

Duplicati is a free and open-source backup software to store encrypted backups online For Windows, macOS and Linux (our favorite, yay!).

Similar to the other backup options in the Cookbook, we can use Duplicati to backup all our data-at-rest to a wide variety of locations, including, but not limited to:

Generic endpoints (FTP, SSH, or WebDAV servers)
Cloud storage providers (Amazon S3, BackBlaze B2, etc)
Cloud services (OneDrive, Google Drive, etc)

Note

Since Duplicati itself offers no user authentication, this design secures Duplicati behind Traefik Forward Auth, so that in order to gain access to the Duplicati UI at all, authentication through the mechanism configured in traefik-forward-auth (to GitHub, GitLab, Google, etc) must have already occurred.

Ingredients

*[X] Docker swarm cluster with persistent shared storage * [X] Traefik and Traefik-Forward-Auth configured per design * [X] Credentials for one of the Duplicati's supported upload destinations

Preparation

Setup data locations

We'll need a folder to store a docker-compose configuration file and an associated environment file. If you're following my filesystem layout, create /var/data/config/duplicati (for the config), and /var/data/duplicati (for the metadata) as per the following example:

mkdir /var/data/config/duplicati
mkdir /var/data/duplicati
cd /var/data/config/duplicati

Prepare {{ no such element: dict object['recipe'] }} environment

Generate a random passphrase to use to encrypt your data. Save this somewhere safe, without it you won't be able to restore!
Seriously, save. it. somewhere. safe.
Create duplicati.env, and populate with the following variables (replace "Europe/London" with your appropriate time zone from this list)

PUID=0
PGID=0
TZ=Europe/London
CLI_ARGS= #optional

Excuse me! Why are we running Duplicati as root?

That's a great question! We're running Duplicati as the root user of the host system because we need Duplicati to be able to read files of all the other services no matter which user that service is running as. After all, Duplicati can't backup your exciting stuff if it can't read the files.

{{ no such element: dict object['recipe'] }} Docker Swarm config

Create a docker swarm config file in docker-compose syntax (v3), something like the example below:

Fast-track with premix! 🚀

"Premix" is a git repository which includes necessary docker-compose and env files for all published recipes. This means that you can launch any recipe with just a git pull and a docker stack deploy 👍.

🚀 Update: Premix now includes an ansible playbook, enabling you to deploy an entire stack + recipes, with a single ansible command! (more here)

version: "3"
services:
  duplicati:
    image: lscr.io/linuxserver/duplicati
    env_file: /var/data/config/duplicati/duplicati.env
    deploy:
      replicas: 1
      labels:
        # traefik common
        - traefik.enable=true
        - traefik.docker.network=traefik_public

        # traefikv1
        - traefik.frontend.rule=Host:duplicati.example.com
        - traefik.port=8200
        - traefik.frontend.auth.forward.address=http://traefik-forward-auth:4181
        - traefik.frontend.auth.forward.authResponseHeaders=X-Forwarded-User
        - traefik.frontend.auth.forward.trustForwardHeader=true

        # traefikv2
        - "traefik.http.routers.duplicati.rule=Host(`duplicati.example.com`)"
        - "traefik.http.routers.duplicati.entrypoints=https"
        - "traefik.http.services.duplicati.loadbalancer.server.port=8200" 
        - "traefik.http.routers.duplicati.middlewares=forward-auth"
    volumes:
      - /var/data/config/duplicati:/config
      - /var/data:/source
    ports:
      - 8200:8200
    networks:
      - traefik_public
      - internal

networks:
  traefik_public:
    external: true
  internal:
    driver: overlay
    ipam:
      config:
        - subnet: 172.16.55.0/24

Note

Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See my list here.

Serving

Launch Duplicati stack

Launch the Duplicati stack by running docker stack deploy duplicati -c <path-to-docker-compose.yml>

Create (and verify!) Your First Backup

Once we authenticate through the traefik-forward-auth provider, we can start configuring your backup jobs via the Duplicati UI. All backup and restore job configuration is done through the UI. Be sure to read through the documentation on Creating a new backup job and Restoring files from a backup for information on how to configure those jobs.

Warning

An untested backup is not really a backup at all. Being sure you can succesfully restore files from your backup now could save you lots of heartache later after "something bad" happens.

Tip

Backing up files on a regular basis is going to use a continually-increasing amount of disk space. To help with this, Duplicati offers a "Smart Backup Retention" scheme that will intelligently remove certain backups as they age while still maintaining a comprehensive backup history. You can set that configuration on the "Options" tab of the backup configuration.

Chef's notes 📓

Quote attributed to Mila Kunis ↩
The Duplicati 2 User's Manual contains all the information you'll need to configure backup endpoints, restore jobs, scheduling and advanced properties for your backup jobs. ↩

Did you receive excellent service? Want to compliment the chef? (..and support development of current and future recipes!) Sponsor me on Github / Ko-Fi / Patreon, or see the contribute page for more (free or paid) ways to say thank you! 👏

Employ your chef (engage) 🤝

Is this too much of a geeky PITA? Do you just want results, stat? I do this for a living - I'm a full-time Kubernetes contractor, providing consulting and engineering expertise to businesses needing short-term, short-notice support in the cloud-native space, including AWS/Azure/GKE, Kubernetes, CI/CD and automation.

Learn more about working with me here.

Want to know now when this recipe gets updated, or when future recipes are added? Subscribe to the RSS feed, or leave your email address below, and we'll keep you updated.