Skip to content

Made changes to your CoreDNS deployment / images? You may find kubeadm uncooperative..

Are you trying to join a new control-plane node to a kubeadm-installed cluster, and seeing an error like this?

start version '8916c89e1538ea3941b58847e448a2c6d940c01b8e716b20423d2d8b189d3972' not supported
unable to get list of changes to the configuration.
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.isCoreDNSConfigMapMigrationRequired

You've changed your CoreDNS deployment, haven't you? You're using a custom image, or an image digest, or you're using an admissionwebhook to mutate pods upon recreation?

Here's what it means, and how to work around it...

We use Connaisseur to enforce an internal policy upon our clusters - we don't run any images not signed with cosign.

Why not use sigstore's policy-controller admission controller?

For one, I didn't know it existed before writing this! But having read up on it, here's why I believe that connaisseur is a better choice for our cluster:

connaisseur vs sigstore's policy-controller admission controller

  • Connaisseur can apply to all namespaces by default, and individual namespaces can opt-out
  • Connaisseur can "mutate" manifests, replacing tag-based images with their cosign-verified digest
  • Connaisseur can post slack webhooks to update an ops team re a policy violation, whether in "enforce" or "audit" mode

When kubeadm init instantiates a new control-plane node, it tries to determine which version of CoreDNS is running in the cluster, by directly examining the coredns pods.

Here's what one of my pods looks like:

Image: registry-internal.elpenguino.net/myorg/coredns:v1.8.6@sha256:8916c89e1538ea3941b58847e448a2c6d940c01b8e716b20423d2d8b189d3972

kubeadm doesn't seem to be able to detect that the image above is at v1.8.6, and instead assumes it to be 8916... (the digest).

The error can't be worked-around by ignoring a pre-flight test, since this particular failure happens "post-flight", and causes the entire install process to fail. The only viable solution currently (I'll report this upstream, but it may end up being a "this-is-by-design" issue), is to explicitly prevent connaisseur from meddling with pods in the kube-system namespace, by labelling the namespace with securesystemsengineering.connaisseur/webhook=ignore.

Aside from the fact that kubeadm could handle this failure more gracefully, I believe that excluding kube-system from admissionwebhooks is a smart move anyway, since kube-system should really be inviolate, and any unexpected changes may interfere with current and future Kubernetes upgrades anyway!