The dry run was green. Every assertion passed, the target version resolved against the apt repo, all three nodes reported Ready, and the playbook printed a line telling me in plain words that it had changed nothing and I should re-run with -e upgrade_confirm=true to do the real thing. So I did. About twenty minutes later my control-plane node was running the new version and sitting cordoned, both workers were still on the old one, and Ansible had stopped on a jsonpath error that had nothing to do with Kubernetes at all.

This is the story of a patch upgrade that half-worked, why the dry run couldn’t have warned me, and the two-line change that means the next one won’t do this.

The setup

The cluster is three Intel NUCs running kubeadm-bootstrapped Kubernetes, the same box I rebuilt drift-free earlier this year. Cilium does the CNI work and replaces kube-proxy entirely, so there is no kube-proxy DaemonSet and nothing should ever recreate one. Longhorn provides storage on the two workers, two replicas per volume, one copy on each worker. The master carries etcd and the control plane and no application data.

Patch bumps within a minor get their own Ansible playbook, upgrade.yml, kept deliberately separate from the build playbook so the destructive teardown role can never fire by accident. It snapshots etcd, upgrades the control plane, then walks the workers one at a time, draining each before touching it. Every mutating task lives inside a when: upgrade_confirm | bool block. Run it plain and you get preflight checks and nothing else; add -e upgrade_confirm=true and it actually moves.

Going from 1.36.1 to 1.36.2 sounds like the most boring thing you can do to a cluster. The apt package is 1.36.2-2.1, incidentally, not the -1.1 you’d guess; the playbook asserts the exact string exists via apt-cache madison before it changes anything, which is the one preflight check that has already earned its keep on a previous bump.

Why the dry run can’t see the part that matters

Here’s the shape of the problem, and it’s worth sitting with because it generalises. A dry run of this playbook sets upgrade_confirm to false, so it runs the preflight and skips every gated task. That is exactly what a dry run is supposed to do; it’s honest about mutation. It is also, by construction, silent about whether any of the mutating tasks are correct, because it never runs them. The tasks that broke were the tasks the check mode was designed never to touch. Green means “I didn’t change anything,” not “this would work.”

I knew that in the abstract. I did not feel it until a real run taught me.

The jsonpath that older kubectl forgave

The failing task was “Wait for the control-plane node to be Ready.” It polls with this:

kubectl get node k8s-master -o jsonpath={.status.conditions[?(@.type=="Ready")].status}

Ansible’s command module doesn’t run a shell; it splits the string with shlex, and shlex quietly eats the double quotes around Ready. So what kubectl actually receives is @.type==Ready, with Ready as a bare identifier. Older kubectl tolerated that. The freshly-installed 1.36.2 client does not:

error: error executing jsonpath "{.status.conditions[?(@.type==Ready)].status}":
Error executing template: unrecognized identifier Ready

Thirty retries, ten seconds apart, each one failing on a parse error rather than on the node’s actual condition. The node was Ready the whole time; the dumped object in the error even shows "type":"Ready","status":"True". The check couldn’t read the answer it already had.

The cruel bit of timing is where in the play it sat. The task runs after kubeadm upgrade apply, after the drain, after the kubelet and kubectl packages go to 1.36.2, and before the uncordon. So the failure left the master fully upgraded, healthy, and cordoned, with the workers untouched behind it. A version-skewed, half-drained cluster is a supported state for kubeadm, but it is not a state you want to discover from a stack trace.

The recovery was calm, which is the point

Nothing was lost, and the partial state was legible, which is the whole reason to build these things as staged plays rather than one big script. Before doing anything I checked whether the master upgrade had actually done its job or just claimed to. The apiserver was bound to 192.168.1.20:6443 and not to :::6443, which matters because kubeadm upgrade apply regenerates the static pod manifests and wipes the IPv4 bind every time; the playbook re-applies it, and here I could confirm that re-apply had held. The kubelet had its address: line. The kube-proxy DaemonSet was still absent. Control-plane pods had restarted and come back. So I uncordoned the master by hand, which is the single step the playbook would have done next if it had got there.

The etcd snapshot from the start of the run, all 23 megabytes of it, was sitting in _scratch where Play 0 had fetched it. I didn’t need it. Having it anyway is the correct amount of etcd snapshot to have before an upgrade.

The second failure was the good kind

I fixed the jsonpath, then re-ran scoped to just the workers. worker1 upgraded cleanly: drain, packages, kubeadm upgrade node, IPv4 rebind, wait, uncordon, fifteen tasks, no drama. worker2 failed on the very first task.

That first task is the Longhorn drain gate, and it had done precisely what I built it to do. Draining worker1 had moved its pods and left its Longhorn replica rebuilding; for a short window the two-replica volumes were down to one good copy and reported degraded. The gate checks for exactly that and refuses to drain the second worker into it, because draining both replica holders in quick succession is how you turn a rolling upgrade into an outage. I would much rather it be too cautious than not cautious enough.

It was cautious in slightly the wrong shape, though. The check was an instant assert, pass-or-fail on a single reading, when the honest thing is a wait: the pool would be healthy again inside a minute. So I let it settle and re-ran worker2 on its own, and it went through like the first.

What changes for next time

Two edits, both small, both born from watching the thing fail.

Readiness no longer polls jsonpath at all. It uses kubectl wait --for=condition=Ready node/<name> --timeout=300s, which needs no quoting, doesn’t care which kubectl minor is installed, and exists for exactly this purpose. The Longhorn gate keeps its check but drops the quoted jsonpath for a flat {.items[*].status.robustness} that shlex can’t damage, and it now retries for up to five minutes rather than failing on the first reading; a single run will walk both workers again, waiting out each rebuild on its own.

The larger lesson doesn’t have a two-line fix, so I wrote it into the runbook instead: this playbook’s dry run cannot catch a bug in a gated task, and pretending otherwise is how you end up trusting a green check that never ran the code. The real answer is probably a smoke path that exercises the command shapes without mutating a cluster. For now, honesty in the docs will do.

There’s one more thing on the list, older than this upgrade. The IPv4 rebind is a patch over kubeadm regenerating manifests on every apply; the proper fix is to move --bind-address into the kubeadm ClusterConfiguration so kubeadm renders it itself and there’s nothing to wipe. Do that and the rebind task disappears, and so does the small window where a stray out-of-band kubeadm upgrade could quietly put the apiserver back on dual-stack. It’s tracked. It wasn’t going to happen at eleven at night mid-upgrade.

Where it landed

All three nodes are on v1.36.2 now, confirmed both from the API and by sshing in and asking the binaries directly; kubeadm, kubelet and kubectl all report 1.36.2 and all three sit under apt-mark hold. kube-proxy is still absent. The apiserver is still IPv4-only. Longhorn is healthy, and the logs dashboard and Ollama both answer 200 through the gateway. The upgrade worked. The playbook that ran it had two bugs that only a real run could surface, and it has neither of them now, which is the most you can ask of a first run: that it fails safely, tells you why, and leaves the tooling better than it found it.