NFS-Subdirectory Provider on Kubernetes
This storage provider allows you to use an NFS server like a native K8s storage provider, letting you to use mass storage for things like media or other large files. Why would this be useful? Things that you don't want to be replicated, for example, media (replicating 4.5TB can get expensive quick) or large data such as game servers! Of course, this does add a singe point of failure, but a lot less expensive than replicating data out to many nodes.
NFS-Subdirectory Provider requirements
- An already existing NFS server
SQLite hates NFS
This recpie assumes you have an NFS server ready to go with a username and a password. Setting this up is outside the current scope of this recipe. This provider is also not to be used for persisting SQLite databases, as storing them on NFS will cause the database to corrupt.
We're going to install a helm chart from the NFS Subdirectory External Provisioner chart repository, so I create the following in my flux repo:
apiVersion: source.toolkit.fluxcd.io/v1beta1 kind: HelmRepository metadata: name: nfs-subdir namespace: flux-system spec: interval: 15m url: https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
Note that I have shortened the name to nfs-subdir, a theme you will find running throughout.1
We need a namespace to deploy our HelmRelease and associated ConfigMaps into. Per the flux design, I create this example yaml in my flux repo at
apiVersion: v1 kind: Namespace metadata: name: nfs-subdir
Now that the "global" elements of this deployment have been defined, we do some "flux-ception", and go one layer deeper, adding another Kustomization, telling flux to deploy any YAMLs found in the repo at
/nfs-subdir. I create this example Kustomization in my flux repo:
apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: nfs-subdir namespace: flux-system spec: interval: 15m path: nfs-subdir prune: true # remove any elements later removed from the above path timeout: 2m # if not set, this defaults to interval duration, which is 1h sourceRef: kind: GitRepository name: flux-system healthChecks: - apiVersion: apps/v1 kind: Deployment name: nfs-subdir-external-provisioner namespace: nfs-suibdir
Now we're into the nfs-subdir-specific YAMLs. First, we create a ConfigMap, containing the entire contents of the helm chart's values.yaml. Paste the values into a
values.yaml key as illustrated below, indented 4 spaces (since they're "encapsulated" within the ConfigMap YAML). I create this example yaml in my flux repo:
apiVersion: v1 kind: ConfigMap metadata: name: nfs-subdir-helm-chart-value-overrides namespace: nfs-subdir data: values.yaml: |- # (1)! # <upstream values go here>
- Paste in the contents of the upstream
values.yamlhere, intended 4 spaces, and then change the values you need as illustrated below.
Values you will want to change from the default are:
nfs: server: # (1)! path: # (2)! mountOptions: # (3)!
- Insert server IP or DNS name
- Insert mount path here
- Set things like your user or specific versions here
Finally, having set the scene above, we define the HelmRelease which will actually deploy the provider into the cluster. I save this in my flux repo:
apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: nfs-subdir namespace: nfs-subdir spec: chart: spec: chart: nfs-subdir-external-provisioner version: 4.X.X sourceRef: kind: HelmRepository name: nfs-subdir namespace: flux-system interval: 15m timeout: 5m releaseName: nfs-subdir-external-provisioner valuesFrom: - kind: ConfigMap name: nfs-subdir-helm-chart-value-overrides valuesKey: values.yaml # This is the default, but best to be explicit for clarity
Install the provider.
Commit the changes to your flux repository, and either wait for the reconciliation interval, or force a reconcilliation using
flux reconcile source git flux-system. You should see the kustomization appear...
~ ❯ flux get kustomizations | grep nfs-subdir nfs-subdir main@sha1:f1b8c5ad False True Applied revision: main@sha1:f1b8c5ad ~ ❯
The helmrelease should be reconciled...
~ ❯ $ flux get helmreleases -n nfs-subdir NAME REVISION SUSPENDED READY MESSAGE nfs-subdir 4.0.18 False True Release reconciliation succeeded ~ ❯
And you should have a happy NFS-Subdirectory pod:
~ ❯ kubectl get pods -n nfs-subdir NAME READY STATUS RESTARTS AGE nfs-subdir-external-provisioner-9cf9d78b5-6zd7r 1/1 Running 22 (4d11h ago) 105d ~ ❯
You can now use this new provider to use an external NFS server for storage.
How do I know it's working?
So the provisioner is running, but how do we know we can actually provision volumes?
Create a PVC, by running:
cat <<EOF | kubectl create -f - apiVersion: v1 kind: PersistentVolumeClaim metadata: name: nfs-subdir-pvc spec: accessModes: - ReadWriteOnce storageClassName: nfs-client resources: requests: storage: 128Mi EOF
Examine the PVC by running
kubectl describe pvc nfs-subdir-pvc
Now create a pod to consume the PVC, by running:
cat <<EOF | kubectl create -f - apiVersion: v1 kind: Pod metadata: name: nfs-subdir-test spec: containers: - name: volume-test image: nginx:stable-alpine imagePullPolicy: IfNotPresent volumeMounts: - name: nfs-subdir-rocks mountPath: /data ports: - containerPort: 80 volumes: - name: nfs-subdir-rocks persistentVolumeClaim: claimName: nfs-subdir-pvc EOF
Examine the pod by running
kubectl describe pod nfs-subdir-test.
Assuming that the pod is in a
Running state, then nfs-subdir is working!
Clean up by running:
kubectl delete pod nfs-subdir-test kubectl delete pvc nfs-subdir-pvc
What have we achieved? We have a storage provider that can use an NFS server as it's storage backend, useful for large files, such as media for the autopirate recipe!
- We have a new storage provider
Chef's notes 📓
The reason I shortened it is so I didn't have to type nfs-subdirectory-provider each time. If you want that sort of pain in your life, feel free to change it! ↩
Tip your waiter (sponsor) 👏
Did you receive excellent service? Want to compliment the chef? (..and support development of current and future recipes!) Sponsor me on Github / Ko-Fi / Patreon, or see the contribute page for more (free or paid) ways to say thank you! 👏
Employ your chef (engage) 🤝
Is this too much of a geeky PITA? Do you just want results, stat? I do this for a living - I'm a full-time Kubernetes contractor, providing consulting and engineering expertise to businesses needing short-term, short-notice support in the cloud-native space, including AWS/Azure/GKE, Kubernetes, CI/CD and automation.
Learn more about working with me here.
Flirt with waiter (subscribe) 💌
Want to know now when this recipe gets updated, or when future recipes are added? Subscribe to the RSS feed, or leave your email address below, and we'll keep you updated.