TopoLVM on Kubernetes

TopoLVM is like Local Path Provisioner, in that it deals with local volumes specific to each Kubernetes node, but it offers more flexibility, and is more suited for a production deployment.

Additional benefits offered by TopoLVM are:

• Volumes can by dynamically expanded
• The scheduler is capacity-aware, and can schedule pods to nodes with enough capacity for the pods' storage requirements
• Multiple storageclasses are supported, so you could, for example, create a storageclass for HDD-backed volumes, and another for SSD-backed volumes

Preparation

Volume Group

Finally you get to do something on your nodes without YAML or git, like a pre-GitOps, bare-metal-cavemonkey!

On each node, you'll need an LVM Volume Group (VG) for TopoLVM to consume. The most straightforward to to arrange this is to dedicate a disk to TopoLVM, and create a dedicated PV and VG for it.

In brief, assuming /dev/sdb is the disk (and it's unused), you'd do the following to create a VG called VG-topolvm:

pvcreate /dev/sdb
vgcreate VG-topolvm /dev/sdb

Namespace

We need a namespace to deploy our HelmRelease and associated ConfigMaps into. Per the flux design, I create this example yaml in my flux repo:

apiVersion: v1
kind: Namespace
metadata:
  name: topolvm-system

HelmRepository

Next, we need to define a HelmRepository (a repository of helm charts), to which we'll refer when we create the HelmRelease. We only need to do this once per-repository. In this case, we're using the official TopoLVM helm chart, so per the flux design, I create this example yaml in my flux repo:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: topolvm
  namespace: flux-system
spec:
  interval: 15m
  url: https://topolvm.github.io/topolvm

Kustomization

Now that the "global" elements of this deployment (Namespace and HelmRepository) have been defined, we do some "flux-ception", and go one layer deeper, adding another Kustomization, telling flux to deploy any YAMLs found in the repo at /topolvm. I create this example Kustomization in my flux repo:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: topolvm--topolvm-system
  namespace: flux-system
spec:
  interval: 15m
  path: ./topolvm-system
  prune: true # remove any elements later removed from the above path
  timeout: 2m # if not set, this defaults to interval duration, which is 1h
  sourceRef:
    kind: GitRepository
    name: flux-system
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: topolvm-controller
      namespace: topolvm-system
    - apiVersion: apps/v1
      kind: DaemonSet
      name: topolvm-lvmd-0
      namespace: topolvm-system
    - apiVersion: apps/v1
      kind: DaemonSet
      name: topolvm-node
      namespace: topolvm-system
    - apiVersion: apps/v1
      kind: DaemonSet
      name: topolvm-scheduler
      namespace: topolvm-system

ConfigMap

Now we're into the topolvm-specific YAMLs. First, we create a ConfigMap, containing the entire contents of the helm chart's values.yaml. Paste the values into a values.yaml key as illustrated below, indented 4 spaces (since they're "encapsulated" within the ConfigMap YAML). I create this example yaml in my flux repo:

apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: topolvm-helm-chart-value-overrides
  namespace: topolvm-system
data:
  values.yaml: |-
    # paste chart values.yaml (indented) here and alter as required>

Then work your way through the values you pasted, and change any which are specific to your configuration. You might want to start off by changing the following to match the name of the volume group you created above.¹

lvmd:
  # lvmd.managed -- If true, set up lvmd service with DaemonSet.
  managed: true

  # lvmd.socketName -- Specify socketName.
  socketName: /run/topolvm/lvmd.sock

  # lvmd.deviceClasses -- Specify the device-class settings.
  deviceClasses:
    - name: ssd
      volume-group: myvg1
      default: true
      spare-gb: 10

HelmRelease

Lastly, having set the scene above, we define the HelmRelease which will actually deploy TopoLVM into the cluster, with the config we defined above. I save this in my flux repo:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: topolvm
  namespace: topolvm-system
spec:
  chart:
    spec:
      chart: topolvm
      version: 3.x
      sourceRef:
        kind: HelmRepository
        name: topolvm
        namespace: flux-system
  interval: 15m
  timeout: 5m
  releaseName: topolvm
  valuesFrom:
  - kind: ConfigMap
    name: topolvm-helm-chart-value-overrides
    valuesKey: values.yaml # This is the default, but best to be explicit for clarity

Serving

Deploy TopoLVM

Having committed the above to your flux repository, you should shortly see a topolvm kustomization, and in the topolvm-system namespace, a bunch of pods:

demo@shredder:~$ kubectl get pods -n topolvm-system
NAME                                  READY   STATUS    RESTARTS   AGE
topolvm-controller-85698b44dd-65fd9   4/4     Running   0          133m
topolvm-controller-85698b44dd-dmncr   4/4     Running   0          133m
topolvm-lvmd-0-98h4q                  1/1     Running   0          133m
topolvm-lvmd-0-b29t8                  1/1     Running   0          133m
topolvm-lvmd-0-c5vnf                  1/1     Running   0          133m
topolvm-lvmd-0-hmmq5                  1/1     Running   0          133m
topolvm-lvmd-0-zfldv                  1/1     Running   0          133m
topolvm-node-6p4qz                    3/3     Running   0          133m
topolvm-node-7vdgt                    3/3     Running   0          133m
topolvm-node-mlp4x                    3/3     Running   0          133m
topolvm-node-sxtn5                    3/3     Running   0          133m
topolvm-node-xf265                    3/3     Running   0          133m
topolvm-scheduler-jlwsh               1/1     Running   0          133m
topolvm-scheduler-nj8nz               1/1     Running   0          133m
topolvm-scheduler-tg72z               1/1     Running   0          133m
demo@shredder:~$

How do I know it's working?

So the controllers etc are running, but how do we know we can actually provision volumes?

Create PVC

Create a PVC, by running:

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: topolvm-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: topolvm-provisioner
  resources:
    requests:
      storage: 128Mi
EOF

Examine the PVC by running kubectl describe pvc topolvm-pvc

Create Pod

Now create a pod to consume the PVC, by running:

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: topolvm-test
spec:
  containers:
  - name: volume-test
    image: nginx:stable-alpine
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: topolvm-rocks
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: topolvm-rocks
    persistentVolumeClaim:
      claimName: topolvm-pvc
EOF

Examine the pod by running kubectl describe pod topolvm-test.

Clean up

Assuming that the pod is in a Running state, then TopoLVM is working!

Clean up your mess, little bare-metal-cave-monkey , by running:

kubectl delete pod topolvm-test
kubectl delete pvc topolvm-pvc

Troubleshooting

Are things not working as expected? Try one of the following to look for issues:

1. Watch the lvmd logs, by running kubectl logs -f -n topolvm-system -l app.kubernetes.io/name=topolvm-lvmd
2. Watch the node logs, by running kubectl logs -f -n topolvm-system -l app.kubernetes.io/name=topolvm-node
3. Watch the scheduler logs, by running kubectl logs -f -n topolvm-system -l app.kubernetes.io/name=scheduler
4. Watch the controller node logs, by running kubectl logs -f -n topolvm-system -l app.kubernetes.io/name=controller

Chef's notes 📓

Tip your waiter (sponsor) 👏

Did you receive excellent service? Want to compliment the chef? (..and support development of current and future recipes!) Sponsor me on Github / Ko-Fi / Patreon, or see the contribute page for more (free or paid) ways to say thank you! 👏

Employ your chef (engage) 🤝

Is this too much of a geeky PITA? Do you just want results, stat? I do this for a living - I'm a full-time Kubernetes contractor, providing consulting and engineering expertise to businesses needing short-term, short-notice support in the cloud-native space, including AWS/Azure/GKE, Kubernetes, CI/CD and automation.

Learn more about working with me here.

Flirt with waiter (subscribe) 💌

Want to know now when this recipe gets updated, or when future recipes are added? Subscribe to the RSS feed, or leave your email address below, and we'll keep you updated.

Your comments? 💬