Running a spouse-approved Kubernetes homelab

"Spouse approved" is doing a lot of work in the description of my home Kubernetes cluster. It means the things my household actually depends on — home automation, the media library, DNS — keep working even when I'm elbow-deep in an upgrade. That constraint has shaped the cluster more than any technology choice.

The hardware is deliberately boring

The cluster is eight Lenovo Tiny mini-PCs: three i7-6700T control-plane nodes and five i5-6500T workers, each with a SATA SSD for the OS and an NVMe drive that belongs to Ceph. Used corporate minis are cheap, quiet, sip power, and when one dies you buy another for the price of a nice dinner. The interesting parts live in software.

No SSH, no snowflakes

Every node runs Talos Linux — an immutable, API-only Kubernetes OS. There is no shell to log into and no package manager to drift; node configuration is generated by talhelper from a single declarative file in the repo, with secrets encrypted via SOPS and age. OS and Kubernetes upgrades are themselves Kubernetes resources, gated on health checks — an upgrade won't proceed unless Ceph reports HEALTH_OK, backups aren't mid-sync, and every node is Ready. The cluster declines to break itself, which is most of what "spouse approved" means.

GitOps or it didn't happen

Everything above the OS is reconciled by Flux from the Git repository. Nothing gets kubectl apply'd by hand. That buys three things:

  1. Disaster recovery is a re-clone. Bootstrap Talos, point Flux at the repo, and the cluster rebuilds itself. The single unrecoverable item is one age key.
  2. Every change has a commit. When something breaks at 11pm, git log is my incident timeline.
  3. Dependency updates are pull requests. Renovate runs on a weekend schedule and opens PRs for chart and image updates; CI renders a flux-local diff of exactly what would change and comments it on the PR, kubeconform validates every manifest, and a gitleaks sweep makes sure nothing secret ever lands in history. I review the diff and merge from my phone. The CI itself runs on self-hosted runners… inside the cluster.

Runtime secrets don't live in Git at all — External Secrets Operator pulls them from Bitwarden Secrets Manager, so SOPS is only needed for the handful of bootstrap credentials.

What actually runs on it

The workloads are the point: Home Assistant, Zigbee2MQTT, and Frigate for home automation (fed by dahua-companion, my doorbell-to-MQTT bridge), the full *arr media stack, Paperless, Mealie, CloudNativePG for Postgres, EMQX for MQTT, and a Prometheus/Grafana/Loki observability stack that watches all of it — plus the router, the NAS, and the UPS.

Storage is Rook-Ceph across the workers' NVMe drives, with Volsync and Kopia backing up every PVC — including a canary job that periodically restores a backup to prove the backups are real, and a nightly sync of the backup buckets to storage outside the cluster.

What "spouse approved" actually requires

The rules that emerged the hard way:

  • DNS must survive the cluster. AdGuard Home runs outside Kubernetes; the cluster merely pushes records to it (via a small Go sidecar I wrote to let wildcard and per-service rewrites coexist). If the cluster is down, the internet still works.
  • Power loss is a handled case. The nodes run NUT clients wired to the UPS, so a long outage triggers a clean shutdown instead of a Ceph recovery adventure.
  • Everything critical has a fallback. The lights still have physical switches. If the cluster is down, the house is merely less convenient, never broken.

What I'd tell you to do differently

Start with GitOps from day one — retrofitting it after months of hand-applied manifests is miserable (I know because this is cluster number two; the first one is preserved in home-k8s-cluster-old). Buy identical used mini-PCs instead of one big server. Treat backups as untested until something automatically restores them. And resist the urge to run everything: each workload you host is a small ongoing tax, and the fun of self-hosting comes from the services you keep, not the ones you tried once.