The cluster was about a year old and I could no longer tell you how it had been built. That’s the real reason I tore it down. Three Intel NUCs running kubeadm Kubernetes, Flannel for the network, MetalLB handing out LoadBalancer IPs, kubernetes/ingress-nginx out front; all of it stood up by hand over a few evenings and never written down anywhere. The symptom that finally pushed me was small and stupid: an orphaned ingress-nginx-lb Service that had been sitting <pending> for 334 days. Nothing depended on it. I just couldn’t remember why it existed, and that bothered me more than it should have.

So this wasn’t a migration. It was a deliberate, drift-free reinstall on the same hardware, with every decision captured as either an Ansible playbook or a declarative manifest in git. If I have to do it again in a year, I want to run one command and walk away.

What I started with

Three NUCs, all on Ubuntu 24.04.4 LTS, asymmetric on RAM which matters later:

Node IP RAM Role Model
k8s-master 192.168.1.20 16 GB control-plane NUC7i3BNK
k8s-worker1 192.168.1.21 32 GB worker NUC7i5DNB
k8s-worker2 192.168.1.22 32 GB worker NUC7i5DNB

The old stack was Kubernetes v1.33.8 (with a v1.33.1 to v1.33.8 control-plane skew nobody asked for), Flannel CNI, MetalLB on pool 192.168.1.30-39, ingress-nginx, containerd, kube-proxy in IPVS mode. Workloads were nginx, a demo httpd, a web hello-app, and ollama. The one piece of good luck in the whole exercise: zero persistent storage. No PVs, no PVCs, no StorageClasses anywhere. Every workload was stateless, so the migration risk was basically nil; I could nuke the cluster without losing a byte.

That last fact is what made a clean rebuild sane rather than reckless.

The target

Same three NUCs, same Ubuntu, but a very different software stack and one big new capability (storage). Here’s the shape of it.

flowchart TB
  subgraph host[Host: Ubuntu 24.04 + kubeadm, brought up by Ansible]
    direction TB
    subgraph net[Networking - Cilium v1.19.4]
      cni[CNI + kube-proxy replacement]
      lb[L2 announcements + LB-IPAM<br>pool 192.168.1.30-39]
    end
    subgraph ns[North-south - Traefik v3.7.1]
      gw[Gateway API: one Gateway, two listeners<br>:80 redirect, :443 wildcard TLS]
    end
    subgraph tls[TLS - cert-manager v1.20.2]
      cm[wildcard *.jeakyl.com<br>Let's Encrypt via Route53 DNS-01]
    end
    subgraph store[Storage - two tiers]
      lh[Longhorn v1.11.2<br>RWO on worker NVMe, 2 replicas, default SC]
      syn[Synology CSI v1.3.0<br>RWO iSCSI off the NAS]
    end
  end
  classDef start fill:#1b2740,color:#fff,stroke:#1b2740;

Read that top to bottom and you have the whole post. Cilium replaces both Flannel and MetalLB: it does the CNI, replaces kube-proxy entirely, and announces LoadBalancer IPs over L2. Traefik v3 in Gateway API mode replaces ingress-nginx, which went end-of-life in late 2025; apps declare HTTPRoute resources instead of Ingress. cert-manager issues a single wildcard *.jeakyl.com from Let’s Encrypt over a Route53 DNS-01 challenge. And storage, the genuinely new bit, is two tiers: Longhorn for fast local block on the workers’ unused NVMe drives, Synology CSI for iSCSI LUNs off the NAS.

The Kubernetes jump itself was three minor versions, v1.33 to v1.36.1. kubeadm only supports one minor at a time on an in-place upgrade, but a teardown-and-reinstall skips that constraint cleanly because no cluster state is preserved. Another argument for rebuilding rather than upgrading.

Ansible does the boring half

I had a brief flirtation with Talos Linux for the host OS. It’s a lovely idea, an immutable API-driven Kubernetes appliance, but I deferred it: the NUCs already run Ubuntu, I know Ubuntu, and I didn’t want to learn an immutable-OS workflow on the same weekend I was rebuilding everything else. So the pivot was Ubuntu stays, Ansible drives the bootstrap.

One playbook, ansible/site.yml, with five roles in order: teardown, os-prep, k8s-install, control-plane, worker. The teardown role is the scary one because it’s genuinely destructive – kubeadm reset, apt purge of the old packages, wiping /var/lib/etcd and /etc/kubernetes, flushing iptables and nftables and ipvsadm leftovers. It has to be idempotent so a half-failed run is safe to re-run, and since there was no persistent storage to lose, I could be aggressive about it.

The whole thing runs in ten to fifteen minutes: teardown and OS prep happen on all three nodes in parallel, package install is about three minutes per node in parallel, then kubeadm init runs serially on the master and the two workers join in roughly thirty seconds each. The control-plane role fetches admin.conf straight back to my workstation as the repo-local .kube/config, so the moment the playbook finishes I can talk to the cluster.

A detail worth recording because it cost me a confused minute. If you dry-run the playbook with --check, the Kubernetes package install fails, and that failure is expected:

no available installation candidate for kubeadm=1.36.1-1.1

In check mode Ansible doesn’t actually write the new /etc/apt/sources.list.d/kubernetes.list, so the subsequent apt install queries the old v1.33 repo where no 1.36.x exists. In a real run the file gets written first and the install succeeds. I’d verified kubeadm=1.36.1-1.1 was in the v1.36 repo before pinning it, so I knew the version was real; the dry-run is “successful” as long as everything up to that apt step reports ok or changed. Worth a comment in the runbook so future-me doesn’t panic.

One more thing the rebuild forced: IPv4-only binding throughout. Mixed v4/v6 binding on the NUCs was causing intermittent weirdness, so the playbook and the manifests now pin v4 everywhere rather than letting components pick.

Cilium: one component instead of three

This is the change I’m happiest with. The old cluster had Flannel doing pod networking, MetalLB doing LoadBalancer IPs, and kube-proxy in IPVS mode doing service routing. Cilium does all three. The kubeadm init runs with --skip-phases=addon/kube-proxy so there’s no kube-proxy DaemonSet at all; Cilium’s kubeProxyReplacement: true takes over, which means it needs k8sServiceHost pinned directly to the control-plane IP since there’s no proxy to find the API server through.

kubeProxyReplacement: true
k8sServiceHost: 192.168.1.20
k8sServicePort: 6443
ipam:
  mode: kubernetes
l2announcements:
  enabled: true
externalIPs:
  enabled: true

The LoadBalancer pool carries over the exact MetalLB range so nothing downstream had to change:

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: jeakyl-lan
spec:
  blocks:
    - start: "192.168.1.30"
      stop:  "192.168.1.39"

Note the cilium.io/v2, not v2alpha1. The IP-pool CRD went GA back in Cilium 1.16, and the alpha apiVersion, while still served, is the kind of thing that bites you on an upgrade two years from now. Pin the GA version while you’re touching it.

L2 announcement is what makes a bare-metal LoadBalancer actually work without a BGP router: one node ARP-replies for each assigned IP, and if that node dies another picks it up. The policy matches the LAN interface by regex:

interfaces: ["^en.*"]

I’d confirmed during prep that all three NUCs name their LAN NIC eno1, so ^en.* covers it. There’s no BGP router on my LAN, so L2 was the only sensible option; if I ever put a real router in, BGP would be the upgrade.

North-south: Gateway API, not Ingress

ingress-nginx being end-of-life was the trigger, but Gateway API is the better model regardless. Instead of a pile of Ingress objects each re-declaring TLS and annotations, there’s a single Gateway with two listeners, and apps in any namespace attach HTTPRoute resources to it. Here’s the path a request takes from a browser on my LAN.

flowchart LR
  client[LAN client<br>https://web.jeakyl.com] --> pihole[PiHole<br>resolves to 192.168.1.31]
  pihole --> lbip[192.168.1.31<br>Cilium L2-announced LB IP]
  lbip --> svc[Traefik LoadBalancer Service]
  svc --> listener{Gateway 'jeakyl'<br>which listener?}
  listener -->|":80 HTTP"| redirect[308 redirect to https]
  listener -->|":443 HTTPS"| terminate[terminate wildcard TLS]
  terminate --> route[HTTPRoute match<br>hostname web.jeakyl.com]
  route --> backend[web Service to hello-app pod]
  classDef start fill:#1b2740,color:#fff,stroke:#1b2740;

PiHole is authoritative for the LAN and resolves every *.jeakyl.com name to 192.168.1.31, the Gateway’s LB IP, which Cilium announces over L2. Traefik terminates TLS on the :443 listener using the wildcard cert, then the matching HTTPRoute forwards to the backing Service. The :80 listener does nothing but issue a 308 to HTTPS.

Getting Traefik v3 to behave under the chart took four corrections that the docs do not put in front of you, and I’ll list them because each one cost real time:

  • The chart version that ships Traefik v3.7.1 is 40.2.0, not the 35.x I’d guessed from an older blog post. Always helm search repo traefik/traefik --versions rather than trusting a number you remember.
  • Entrypoints have to bind :80 and :443 directly. Traefik’s Gateway provider matches a listener’s port to an entrypoint of the same port number, so the chart-default 8000/8443 left both listeners stuck PortUnavailable. Binding low ports as non-root then needs NET_BIND_SERVICE added back to an otherwise drop-everything securityContext.
  • The HTTP-to-HTTPS redirect has to happen at the entrypoint, not via a filter-only HTTPRoute. Traefik v3.7.1’s Gateway provider doesn’t create a router for a redirect-only route, so my tidy little RequestRedirect HTTPRoute just 404’d. I deleted it and set redirections.entryPoint on the :80 port instead.
  • The dashboard needs api.insecure: true so the HTTPRoute can backendRef the traefik:8080 service. With insecure: false the dashboard is only reachable through a router to api@internal, which Gateway API can’t point a backendRef at. The tradeoff is no dashboard auth and it’s also reachable on the LAN at 192.168.1.31:8080; acceptable on a home network, not something I’d do anywhere real.

The Gateway itself is the clean part. One object, two listeners, TLS referencing a Secret in the same namespace so I avoid a cross-namespace ReferenceGrant:

spec:
  gatewayClassName: traefik
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces: { from: All }
    - name: https
      port: 443
      protocol: HTTPS
      hostname: "*.jeakyl.com"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: wildcard-jeakyl-com-tls
      allowedRoutes:
        namespaces: { from: All }

TLS, and the split-horizon trap

cert-manager issues one wildcard *.jeakyl.com (plus the apex) from Let’s Encrypt, solved over DNS-01 against Route53 using an IAM user that already existed from a previous experiment. Staging issuer first, always, to keep clear of Let’s Encrypt’s production rate limits; switch the Certificate’s issuerRef to production only once staging issues cleanly.

The part that will catch anyone running PiHole as their LAN DNS: cert-manager does a propagation self-check before it tells Let’s Encrypt to validate, and that check must not go through PiHole. My PiHole answers jeakyl.com with LAN A records (the whole point of split-horizon), so the public _acme-challenge TXT record would never appear to resolve. The fix is two controller flags forcing the propagation check onto public resolvers:

extraArgs:
  - --dns01-recursive-nameservers=8.8.8.8:53,1.1.1.1:53
  - --dns01-recursive-nameservers-only=true

Without --dns01-recursive-nameservers-only=true it’ll still consult the system resolver and hang. With it, propagation through Route53 takes one to three minutes and the cert goes Ready.

Storage: the genuinely new tier

The old cluster had no storage at all, which is embarrassing for something that ran for a year. Each worker NUC has a 1 TB NVMe SSD doing nothing, and there’s a Synology DS419slim on the LAN. So: two tiers.

flowchart TB
  subgraph pods[Workloads]
    ollama[ollama StatefulSet]
    generic[anything needing a PV]
  end
  subgraph tierA[Longhorn - default StorageClass]
    direction TB
    lha[(NVMe k8s-worker1)]
    lhb[(NVMe k8s-worker2)]
    lha <-->|2 replicas, sync| lhb
  end
  subgraph tierB[Synology CSI]
    iscsi[synology-iscsi RWO LUN]
  end
  nas[(Synology DS419slim<br>NFSv4.0 backup target)]
  generic --> tierA
  ollama --> tierA
  tierA -.nightly backup.-> nas
  pods --> iscsi
  classDef start fill:#1b2740,color:#fff,stroke:#1b2740;

Longhorn is the default StorageClass: RWO block volumes living on the two workers’ NVMe, two replicas (which is the maximum with two workers, and also the point – a synchronous copy on each). It backs up nightly to an NFS share on the Synology. The control-plane NUC’s NVMe is deliberately left out of Longhorn; it’s reserved for the OS and etcd.

Synology CSI provides the second tier. This is where the rebuild stopped being tidy, because four separate things on the NAS side fought me.

The headline disappointment: dynamic RWX NFS doesn’t work on this NAS, and I only found out by trying. Synology CSI provisions NFS shares as DSM shared folders, and shared-folder creation requires a Btrfs volume. My /volume1 is ext4. So CreateVolume fails outright:

Location: /volume1 with ext4 fstype was not supported for
creating smb/nfs protocol's K8s volume.

The cluster therefore has no dynamic RWX StorageClass right now. If I need ReadWriteMany later I’ll either point an nfs-subdir-external-provisioner at a hand-created share, or reformat a NAS volume to Btrfs. For now nothing I run needs RWX, so it stays a documented gap rather than a blocker.

The other three NAS gotchas, briefly, because they’re the sort of thing you’d otherwise rediscover painfully:

  • The DSM service account the driver logs in as, k8s-csi, has to be a member of the administrators group, not merely a storage admin. The CSI driver calls admin-only APIs like SYNO.Core.System; a non-admin logs in fine and then gets error 402, and the driver quietly drops the NAS with “Couldn’t find any host available”. A dedicated account, yes, but it has to be an admin one.
  • multipathd on the nodes has to be masked. It claims Synology iSCSI LUNs as dm-/mpath devices before the CSI driver can mkfs the raw by-path device, and you get “apparently in use by the system” mount failures. The os-prep Ansible role masks it now.
  • Longhorn’s backup driver mounts the target with -t nfs4. My Synology offers NFSv4.0 but not v4.1, and plain v3 fails, so the backup target URL pins ?nfsOptions=nfsvers=4.0.

While I had Longhorn, I fixed the one stateful workload properly. The old ollama Deployment had no volume, so every reschedule re-downloaded the models from scratch. It’s now a StatefulSet with a 50 GiB Longhorn volumeClaimTemplate mounted at /root/.ollama, and the models survive a pod move.

Does it survive a node falling over

The whole point of two replicas and L2 failover is that losing a worker shouldn’t lose anything, so I tested it rather than assuming. Reboot worker1:

ssh -i .ssh/id_k8s root@k8s-worker1.jeakyl.com systemctl reboot

Within about thirty seconds: both LB IPs kept responding because Cilium re-announced .30 and .31 from worker2; Traefik runs two replicas so half the routes were already served from the surviving node; and the ollama pod rescheduled onto worker2, where Longhorn re-attached the surviving replica with no data loss. An arping on 192.168.1.31 showed the MAC had switched to worker2’s eno1, which is exactly the L2 takeover doing its job. When worker1 came back, Longhorn rebuilt the missing replica on its own.

That test is the difference between believing the design works and knowing it does.

What I deliberately didn’t do

A home cluster invites scope creep, so a few explicit non-goals. No HA control plane – the NUCs have asymmetric RAM (16 GB on the master, 32 on the workers), which rules out a sensible stacked-etcd quorum; that waits for a hardware refresh. No GPU; I checked, and none of the NUCs have Thunderbolt 3, so an eGPU isn’t physically possible, and the CPUs are old enough that it’d bottleneck anyway. No service mesh, no mTLS, no BGP, no Talos. And no GitOps yet, though pointing Flux or Argo at the manifests/ directory is the obvious next move now that everything is declarative.

The thing I actually wanted out of this wasn’t any single component. It was the property that the whole cluster now reduces to one Ansible playbook plus a directory of manifests, all in git, all re-runnable. The 334-day <pending> Service is gone, and more to the point, if something like it shows up again I’ll be able to find out why in about a minute.