Titan: a Kubernetes cluster on three Raspberry Pis, running the house

#homelab#kubernetes#self-hosting#raspberry-pi

There’s a well known, sane, reasonable way to self-host things: docker compose on one machine, maybe two if you’re feeling fancy. Files in a folder, docker compose up -d, done by lunch. Plenty of perfectly happy people live this way.

I am not one of them. I run Kubernetes in my house. The cluster is called Titan, a name chosen for being short and sounding vaguely like a spaceship, which is the full extent of the thought that went into it. Three Raspberry Pi 5s, twelve cores and 32GB of RAM between them, and 77 deployments across 45 namespaces behind a proper ingress with real TLS. 115 pods were alive while I wrote this. And I’d do it again.

Here’s the thing: I work with Kubernetes for a living. Production is largely my problem at work, so it’s not like I lack exposure to the real thing. What the home cluster adds is a place where breaking things is free: the blast radius is mine, the lessons are mine, and nobody gets paged. My cluster is a sandbox where I’m allowed to break things in creative ways, learn from the wreckage, and rebuild. That’s the whole point. This post is a tour of what the sandbox looks like right now: the hardware, the storage, the networking, the scale-to-zero tricks, and the quirks I’ve decided to make peace with.

The hardware: one board, three brains

The centrepiece is a DeskPi Super6C, which is exactly what it sounds like if you’ve ever wanted a motherboard for Raspberry Pis. It’s a mini-ITX board that takes up to six Raspberry Pi Compute Modules, gives each one its own M.2 slot, and feeds them all from shared power and networking. You slot in the little compute modules and suddenly you have a cluster that looks like a single quiet box instead of a nest of Ethernet cables and diagonal sticky tape.

Mine is populated with three Pi 5 compute modules:

Each module gets one of the board’s M.2 slots, so each node has a 256GB SSD for real storage, and each compute module brings four Cortex-A76 cores: twelve in total across the chassis. Atlas has the most RAM, which does not buy it VIP treatment: it gets whatever the scheduler throws at it, same as everyone else. The naming theme is astronomy, because at some point every homelab develops a naming scheme and refusing to pick one is apparently not an option.

DeskPi also makes a newer board, the Super4C, which is built specifically for CM5 modules and looks great. Mine runs the CM5s happily on the older board though, and “no reason to upgrade” is a sentence my wallet enjoys hearing.

On top of that, there’s a NAS in the corner doing bulk storage duty, which I’ll get to, because the storage story is where homelabs get interesting.

k3s, not Kubernetes, but also Kubernetes

Running a full control plane across three Pis would eat the RAM budget before a single app got scheduled. So the cluster runs k3s: a certified Kubernetes distribution that ships as a single binary, has its container runtime and networking built in, and is happy on ARM64. microk8s, k0s, and kubeadm all exist, and k3s won here on single binary, ARM64 support, and least ceremony. You get the real Kubernetes API and the full YAML suffering, at a fraction of the overhead. One detail that felt on brand the moment I learned it: in single-server mode, k3s stores the cluster’s entire state in a SQLite database. The control plane’s brain is the same database engine as half the apps it schedules.

Setup is refreshingly boring. One node runs the server install (with the packaged Traefik disabled, more on that later), the other two join as agents with a token. There are two housekeeping steps on a Pi worth knowing about:

• Add cgroup_memory=1 cgroup_enable=memory to the kernel command line, because Kubernetes and memory cgroups need to be on speaking terms.
• Install open-iscsi on every node before deploying Longhorn, unless you enjoy pods stuck in ContainerCreating while Longhorn asks politely for iSCSI.

Upgrades are pinned to specific versions rather than floating on a channel. When the control plane is one node, “upgrade and see what happens” is not a strategy I can afford, so every k3s version bump is a deliberate act.

Storage: small and fast, big and cheap

Two storage systems, two jobs.

Longhorn does the block storage. It takes the local SSD on each node and stitches them into replicated network volumes that follow your pods around. A PVC in one namespace, two replicas on two different nodes, and a node can vanish without the volume being orphaned. Config for the whole thing lives in a tuned manifest rather than the defaults:

• Two replicas instead of three. Three replicas of everything on 768GB of SSD is how you run out of disk for the privilege of durability you mostly don’t need at home. Important data gets covered by backups instead (more below).
• The delete policy is Retain. Deleting a manifest by accident should not delete the data. The PVC can go, the volume stays, and future me says thank you.

The NAS handles everything big. Media libraries, downloads, archive dumps, and the container registry all live on NFS exports, mounted into the cluster through the NFS CSI driver with a nfs-csi storage class. Where Longhorn volumes are small, fast, and replicated, NAS storage is large, cheap, and single copy. The rule of thumb: if it’s app state that would hurt to lose, Longhorn. If it’s gigabytes of stuff that already exists somewhere else, NFS.

The two systems also talk to each other. The NAS is a backup target for some of the Longhorn volumes, and on top of that there’s a cloud storage bucket for the stuff that really needs to survive. So important data gets replicated locally by Longhorn, copied to the NAS, and pushed off-site for the things I never want to explain the absence of. Belt, braces, and a second belt that lives in a different building.

Networking: one wildcard to rule them all

This is the part of the setup I’d genuinely miss if it disappeared.

DNS is handled at my DNS provider: a wildcard record, *.cluster.example.com, points at all three node IPs. Round-robin DNS means any node can answer a request. Each node runs Traefik, so whichever box receives the packet, the gateway is already there to take it.

k3s ships with Traefik out of the box, and I disable it. The honest reason is history: this cluster started out on ingress-nginx, back when that was the default reflex. Then the project announced its retirement, with maintenance winding down in March 2026, and that was my cue to move rather than ride an unpatched controller. Traefik won, mostly for its Gateway API support, and since the bundled k3s Traefik wants the same 80 and 443 ports, the bundled one gets disabled so my own Traefik deployment can take them.

So the cluster runs its own Traefik, configured around the Gateway API instead of classic Ingress resources. One Gateway listener owns the wildcard TLS certificate for *.cluster.example.com, renewed automatically via DNS-01. Because the certificate lives on the listener, every route behind the gateway inherits TLS for free.

The result is that adding a service with a proper HTTPS hostname looks like this:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: myservice
  namespace: myservice
spec:
  parentRefs:
    - name: traefik-gateway
      namespace: traefik
      sectionName: websecure
  hostnames:
    - "myservice.cluster.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: myservice
          port: 80

No cert ceremony, no nginx.conf archaeology. Every service manifest in the repo follows the same shape: namespace, PVC, deployment, service, HTTPRoute. That’s it. There are 38 of these routes live right now, and every one of them is reachable over HTTPS from anything on my network.

The wildcard record is also the quiet hero of every new deployment. Because *.cluster.example.com already points at the nodes, any new service that follows the naming schema is routable the second its manifest is applied. No DNS entry to add anywhere, no waiting for propagation. Apply a manifest, make a request, get an answer.

Scale to zero: the KEDA trick

Here’s the problem with a 32GB cluster: some services are only used a few times a month, but want a real chunk of resources when they do run. A full Linux desktop in the browser, a local AI chat frontend, a text-to-speech engine, the document scanner. Running all of that 24/7 on Pis would mean spending the RAM budget on things that are idle 99% of the time.

Enter KEDA’s HTTP add-on. Sixteen services on the cluster are wired to a HTTPScaledObject, which means they sit at zero replicas while nobody is using them. Requests for those hosts get parked at the KEDA interceptor, which counts the incoming requests, scales the target deployment from zero to one, and hands the traffic over once the pod exists. After about ninety minutes of silence, it scales back down. Cold starts vary from a few seconds to tens of seconds depending on the service, and generous startup probes are what make that survivable: Kubernetes waits appropriately instead of declaring the pod dead mid-boot. The always-on footprint of those sixteen services is zero pods; they cost RAM only while in use.

The pattern is the same for all of them: “I want this available on demand, I do not want it resident.” The full roll call, including which services sit at zero, is in the table further down.

The honest caveats: it only works for request-driven services. Anything with background jobs, cron-like behaviour, or persistent connections needs to stay always-on. And the cross-namespace wiring between a route and the KEDA interceptor needs a ReferenceGrant, which is one of those Kubernetes ceremony things that fails with an error message you will only look up once, then remember forever.

Getting in from the outside: Tailscale

A wildcard DNS record pointing at your machines is great for the LAN, but what about when you’re not home? Two answers, both powered by the Tailscale Kubernetes operator.

Funnel handles the small set of services that should be reachable from anywhere, publicly. Six of them today: my analytics instance, a tools page, a speed test, and a couple of little “hello” pages including one built for a mate. The operator watches for a specific ingress annotation, spins up a proxy pod, registers it with the tailnet, and sorts out TLS automatically. Public hostname, public cert, zero open ports on my router, because traffic arrives through Tailscale’s relays. The trade-off worth knowing: each Funnel ingress gets a dedicated proxy pod, so each one is a single point of failure. For the handful of low-stakes things I publish, that’s a fine trade.

For tailnet-only services, the same operator runs a ProxyGroup: three replicas of ingress proxies, so access from my own devices is highly available rather than one-pod-deep.

The Connector is my favourite piece. It’s a custom resource that runs inside the cluster and gives the tailnet two abilities: it acts as an exit node, so my laptop or phone can route all traffic out through my home connection, and it acts as a subnet router advertising my home LAN range. Together that means being out and about and still reaching the printer, the NAS, the router admin page, anything on the physical network. The cluster has effectively become the front door to the whole house.

Security, briefly

Homelab posts tend to get judged on this part, so here’s the honest version.

The boundary is the router. The wildcard record points at private addresses, which means every *.cluster.example.com hostname simply stops existing outside my LAN. There’s nothing to port-forward to, and no direct path in from the internet at all. The public surface is exactly six services, published through Funnel, and they’re the low-stakes kind: analytics, a speed test, a couple of hello pages.

Getting in from the outside means being on the tailnet, where access is device-level and Tailscale ACLs decide which devices can reach which services.

Secrets never live in git. The manifests reference secrets by name, and an External Secrets Operator deployment pulls the actual values from a cloud secret manager at runtime.

And the fully honest bit: the database admin panels run with no authentication of their own. They exist for the playground databases I spin up while testing things. Anything holding real data, or even remotely sensitive data, lives in a managed service somewhere else entirely. So the LAN boundary is doing all the work for those panels, and that’s a trade I’m comfortable with in a lab. Moving them tailnet-only is on the someday list.

The whole picture

Putting the entry paths on one diagram, because words like “round-robin” get blurry fast:

Three ways in. From the LAN, the wildcard record fans a request out to the Traefik Gateway on whichever node answers, and the HTTPRoute takes it from there. From the internet, Funnel relays traffic to a proxy pod that fronts the handful of public services. And when a request targets a scale-to-zero host, the KEDA interceptor wakes the service up first, then hands the request over once it exists.

Git, registry, and CI that runs on what it builds

The cluster also hosts its own development platform, and this is the part that makes everything else feel real.

Forgejo is the git forge: repos, issues, and a built-in OCI container registry whose data lives on the NAS over NFS. Images built on the cluster get pushed there, and other namespaces pull from it, opt-in per namespace via image pull secrets. Self-hosting the registry means image builds and image pulls happen on the same LAN, which is both fast and pleasingly self-contained. A chunk of the repos also replicate out to a personal GitLab instance, so the code I care about exists in more than one place. Redundancy for the things that matter is cheap when the second copy is just a push mirror.

The Forgejo Actions runners are where it gets fun. Instead of an always-on runner burning RAM for the privilege of idling, they run as a KEDA ScaledJob. A small exporter polls the Forgejo API for pending runs and publishes the count to Prometheus, KEDA watches that metric, and a runner pod appears when there’s work and disappears when there isn’t. Zero to one runner, on demand, driven by a metrics pipeline. Those runners were my stepping stone to Woodpecker, and they stay deployed because scale-to-zero makes idling cost literally nothing: zero pods, zero RAM, zero regrets.

Woodpecker CI handles pipeline builds, and its Kubernetes backend is a genuinely nice fit. The server and a single agent run on the cluster; every pipeline step becomes its own pod with its own image and its own resource requests. Steps that need Docker-in-Docker get it as a sibling service pod rather than nested inside the build, which sidesteps a whole genre of overlay filesystem pain. Steps share a workspace through an RWX Longhorn volume, so the filesystem handoff works even when steps land on different nodes.

The proof is meta: this blog is a Forgejo repo. Push to main, Woodpecker picks it up, builds the Astro site on the cluster, and ships it to Cloudflare Pages. The pipeline that deploys my writing runs on the cluster that the writing describes. At some point I’ll publish this post and the snake will finish eating its tail.

The services, briefly

A quick roll call of what’s actually running. The last column is the interesting one: sixteen of these only exist while someone is using them.

There’s also a small stack of media automation tools I’ve deliberately left out of this post. It runs like everything else here, it stays on the LAN, and it’s better documented elsewhere. That’s the whole story.

The quirks, honestly

Now the part that would make a platform engineer twitch. I know about all of these. They’re deliberate or at least accepted.

SQLite is the load-bearing database. A huge share of self-hosted apps are built around single-instance operation and a SQLite file. That runs beautifully on Kubernetes with one replica. What it doesn’t do is high availability. You cannot casually run two replicas of an app whose entire state is one database file, because the moment both pods decide to write, you’re in a corruption speedrun. So these services are replicas: 1 with a Recreate strategy, and the data lives on Longhorn with Retain semantics. Single instance, but movable and survivable.

The control plane is not highly available. One server node. If titan reboots, workloads keep chugging but nothing new gets scheduled and the API is out until it’s back. A three-node HA control plane is on the someday list, right after “reasons to buy more compute modules”.

It’s all one board. Every node hangs off the same Super6C, which means shared power and shared networking. The individual nodes reboot independently, Kubernetes reschedules things, and life is good, right up until the power flickers and the whole cluster goes down in perfect unison. One chassis is also one firmware update away from all three nodes noticing at once. This is the part the diagram earlier makes obvious and a shopping list hides.

Slow hardware demands patience in the manifests. Startup probes are generous, because some containers just start slowly, and on a Pi “slowly” becomes “geologically”. Without a lenient startup probe, the kubelet decides the app is dead mid-boot and murder-restarts it, forever. Resource requests exist on everything, memory limits on the things that could misbehave, and 21 VerticalPodAutoscalers quietly right-size the always-on services so I don’t have to guess at MiB values ever again.

None of this is “production”. That’s the point. A homelab that never breaks teaches you nothing, and one that breaks in ways you fully understand is a fantastic teacher. But it’s worth saying plainly: if your goal is running services rather than learning the machinery, docker compose on one box is the right answer, as long as your backups are sorted. I chose the other answer on purpose.

Things worth taking with you

Things worth taking no matter what you run:

• Wildcard DNS beats per-service DNS. One record, and every future service is routable the day it’s deployed.
• Two replicas and real backups beat three replicas and none.
• Scale to zero by default for anything request-driven. Always-on is a privilege a service earns by having background jobs, cron, or persistent connections.
• VPA the single-instance stuff you can’t size by gut feel.
• Don’t fight SQLite for high availability. One replica, Recreate strategy, Retain on the volume, move on.
• Pin versions. “Latest” is a gambling term, not a release channel.

What’s next

The roadmap is short and honest: maybe an HA control plane someday, better automated restore testing for the Longhorn backups, and whatever the next “wouldn’t it be funny if” idea turns out to be. That last category is the one with the strongest track record.

If you’ve read this far and you run Kubernetes at work, my pitch is simple: run it at home too, at whatever scale you find interesting. The failure modes are different when you’re also the on-call rotation, and you learn things at home that production would never dare teach you. Preferably not at 2am. Preferably.