Skip to content

AutoPirate

Once the cutting edge of the "internet" (pre-world-wide-web and mosiac days), Usenet is now a murky, geeky alternative to torrents for file-sharing. However, it's cool geeky, especially if you're into having a fully automated media platform.

A good starter for the usenet scene is https://www.reddit.com/r/usenet/. Because it's so damn complicated, a host of automated tools exist to automate the process of finding, downloading, and managing content. The tools included in this recipe are as per the following example:

Autopirate Screenshot

This recipe presents a method to combine these tools into a single swarm deployment, and make them available securely.

Tools included in the AutoPirate stack are:

  • SABnzbd is the workhorse. It takes .nzb files as input (manually or from Sonarr, Radarr, etc), then connects to your chosen Usenet provider, downloads all the individual binaries referenced by the .nzb, and then tests/repairs/combines/uncompresses them all into the final result - media files, to be consumed by Plex, Emby, Komga, Calibre-Web), etc.

  • NZBGet downloads data from usenet servers based on .nzb definitions. Like SABnzbd, but written in C++ and designed with performance in mind to achieve maximum download speed by using very little system resources (this is a popular alternative to SABnzbd)

  • RTorrent is a popular CLI-based bittorrent client, and ruTorrent is a powerful web interface for rtorrent. (Yes, it's not Usenet, but Sonarr/Radarr will let fulfill your watchlist using either Usenet or torrents, so it's worth including)

  • NZBHydra is a meta search for NZB indexers. It provides easy access to a number of raw and newznab based indexers. You can search all your indexers from one place and use it as indexer source for tools like Sonarr or Radarr.

  • Sonarr finds, downloads and manages TV shows

  • Radarr finds, downloads and manages movies

  • Readarr finds, downloads, and manages eBooks

  • Lidarr is an automated music downloader for NZB and Torrent. It performs the same function as Headphones, but is written using the same(ish) codebase as Radarr and Sonarr. It's blazingly fast, and includes beautiful album/artist art. Lidarr supports SABnzbd, NZBGet, Transmission, Β΅Torrent, Deluge and Blackhole (just like Sonarr / Radarr)

  • Mylar is a tool for downloading and managing digital comic books / "graphic novels"

  • Headphones is an automated music downloader for NZB and Torrent, written in Python. It supports SABnzbd, NZBget, Transmission, Β΅Torrent, Deluge and Blackhole.

  • Lazy Librarian is a tool to follow authors and grab metadata for all your digital reading needs. It uses a combination of Goodreads Librarything and optionally GoogleBooks as sources for author info and book info.

  • Ombi provides an interface to request additions to a Plex/Emby/Jellyfin library using the above tools

  • Jackett works as a proxy server: it translates queries from apps (Sonarr, Radarr, Mylar, etc) into tracker-site-specific http queries, parses the html response, then sends results back to the requesting software.

Since this recipe is so long, and so many of the tools are optional to the final result (i.e., if you're not interested in comics, you won't want Mylar), I've described each individual tool on its own sub-recipe page (below), even though most of them are deployed very similarly.

Ingredients

Ingredients

Already deployed:

Related:

  • Traefik Forward Auth to secure your Traefik-exposed services with an additional layer of authentication

Preparation

Setup data locations

We'll need a unique directories for each tool in the stack, bind-mounted into our containers, so create them upfront, in /var/data/autopirate:

mkdir /var/data/autopirate
cd /var/data/autopirate
mkdir -p {lazylibrarian,mylar,ombi,sonarr,radarr,headphones,plexpy,nzbhydra,sabnzbd,nzbget,rtorrent,jackett}

Create a directory for the storage of your downloaded media, i.e., something like:

mkdir /var/data/media

Create a user to "own" the above directories, and note the uid and gid of the created user. You'll need to specify the UID/GID in the environment variables passed to the container (in the example below, I used 4242 - twice the meaning of life).

Secure public access

What you'll quickly notice about this recipe is that every web interface is protected by an OAuth proxy.

Why? Because these tools are developed by a handful of volunteer developers who are focused on adding features, not necessarily implementing robust security. Most users wouldn't expose these tools directly to the internet, so the tools have rudimentary (if any) access control.

To mitigate the risk associated with public exposure of these tools (you're on your smartphone and you want to add a movie to your watchlist, what do you do, hotshot?), in order to gain access to each tool you'll first need to authenticate against your given OAuth provider.

This is tedious, but you only have to do it once. Each tool (Sonarr, Radarr, etc) to be protected by an OAuth proxy, requires unique configuration. I use github to provide my oauth, giving each tool a unique logo while I'm at it (make up your own random string for OAUTH2PROXYCOOKIE_SECRET)

For each tool, create /var/data/autopirate/<tool>.env, and set the following:

OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
PUID=4242
PGID=4242

Create at least /var/data/autopirate/authenticated-emails.txt, containing at least your own email address with your OAuth provider. If you wanted to grant access to a specific tool to other users, you'd need a unique authenticated-emails-<tool>.txt which included both normal email address as well as any addresses to be granted tool-specific access.

Setup components

Stack basics

Start with a swarm config file in docker-compose syntax, like this:

version: '3'

services:

And end with a stanza like this:

networks:
  traefik_public:
    external: true
  internal:
    driver: overlay
    ipam:
      config:
        - subnet: 172.16.11.0/24

Note

Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See my list here.

Assemble more tools..

Continue through the list of tools below, adding whichever tools your want to use, and finishing with the end section:

Chef's notes πŸ““

  1. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition. ↩

Tip your waiter (sponsor) πŸ‘

Did you receive excellent service? Want to compliment the chef? (..and support development of current and future recipes!) Sponsor me on Github / Ko-Fi / Patreon, or see the contribute page for more (free or paid) ways to say thank you! πŸ‘

Employ your chef (engage) 🀝

Is this too much of a geeky PITA? Do you just want results, stat? I do this for a living - I'm a full-time Kubernetes contractor, providing consulting and engineering expertise to businesses needing short-term, short-notice support in the cloud-native space, including AWS/Azure/GKE, Kubernetes, CI/CD and automation.

Learn more about working with me here.

Flirt with waiter (subscribe) πŸ’Œ

Want to know now when this recipe gets updated, or when future recipes are added? Subscribe to the RSS feed, or leave your email address below, and we'll keep you updated.

Your comments? πŸ’¬