Once the cutting edge of the "internet" (pre-world-wide-web and mosiac days), Usenet is now a murky, geeky alternative to torrents for file-sharing. However, it's cool geeky, especially if you're into having a fully automated media platform.
A good starter for the usenet scene is https://www.reddit.com/r/usenet/. Because it's so damn complicated, a host of automated tools exist to automate the process of finding, downloading, and managing content. The tools included in this recipe are as follows:
This recipe presents a method to combine these tools into a single swarm deployment, and make them available securely.
Tools included in the AutoPirate stack are:
- SABnzbd : downloads data from usenet servers based on .nzb definitions
- NZBGet : downloads data from usenet servers based on .nzb definitions, but written in C++ and designed with performance in mind to achieve maximum download speed by using very little system resources (this is a popular alternative to SABnzbd)
- RTorrent is a CLI-based torrent client, which when combined with ruTorrent becomes a powerful and fully browser-managed torrent client. (Yes, it's not Usenet, but Sonarr/Radarr will let fulfill your watchlist using either Usenet or torrents, so it's worth including)
- NZBHydra : acts as a "meta-indexer", so that your downloading tools (radarr, sonarr, etc) only need to be setup for a single indexes. Also produces interesting stats on indexers, which helps when evaluating which indexers are performing well.
- NZBHydra2 : is a high-performance rewrite of the original NZBHydra, with extra features. While still in beta, this NZBHydra2 will eventually supercede NZBHydra
- Sonarr : finds, downloads and manages TV shows
- Radarr : finds, downloads and manages movies
- Mylar : finds, downloads and manages comic books
- Headphones : finds, downloads and manages music
- Lazy Librarian : finds, downloads and manages ebooks
- Ombi : provides an interface to request additions to a Plex/Emby library using the above tools
- Jackett : Provides an local, caching, API-based interface to torrent trackers, simplifying the way your tools search for torrents.
Since this recipe is so long, and so many of the tools are optional to the final result (i.e., if you're not interested in comics, you won't want Mylar), I've described each individual tool on its own sub-recipe page (below), even though most of them are deployed very similarly.
- Docker swarm cluster with persistent shared storage
- Traefik configured per design
- Access to NZB indexers and Usenet servers
- DNS entries configured for each of the NZB tools in this recipe that you want to use
Setup data locations¶
We'll need a unique directories for each tool in the stack, bind-mounted into our containers, so create them upfront, in /var/data/autopirate:
1 2 3
Create a directory for the storage of your downloaded media, i.e., something like:
Create a user to "own" the above directories, and note the uid and gid of the created user. You'll need to specify the UID/GID in the environment variables passed to the container (in the example below, I used 4242 - twice the meaning of life).
Secure public access¶
What you'll quickly notice about this recipe is that every web interface is protected by an OAuth proxy.
Why? Because these tools are developed by a handful of volunteer developers who are focused on adding features, not necessarily implementing robust security. Most users wouldn't expose these tools directly to the internet, so the tools have rudimentary (if any) access control.
To mitigate the risk associated with public exposure of these tools (you're on your smartphone and you want to add a movie to your watchlist, what do you do, hotshot?), in order to gain access to each tool you'll first need to authenticate against your given OAuth provider.
This is tedious, but you only have to do it once. Each tool (Sonarr, Radarr, etc) to be protected by an OAuth proxy, requires unique configuration. I use github to provide my oauth, giving each tool a unique logo while I'm at it (make up your own random string for OAUTH2PROXYCOOKIE_SECRET)
For each tool, create /var/data/autopirate/
1 2 3 4 5
Create at least /var/data/autopirate/authenticated-emails.txt, containing at least your own email address with your OAuth provider. If you wanted to grant access to a specific tool to other users, you'd need a unique authenticated-emails-
Start with a swarm config file in docker-compose syntax, like this:
1 2 3
And end with a stanza like this:
1 2 3 4 5 6 7 8
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See my list here.
Assemble the tools..¶
Now work your way through the list of tools below, adding whichever tools your want to use, and finishing with the end section:
- Lazy Librarian
- End (launch the stack)
Tip your waiter (support me) 👏¶
Did you receive excellent service? Want to make your waiter happy? (..and support development of current and future recipes!) See the support page for (free or paid) ways to say thank you! 👏