Owning the build: Forgejo, Woodpecker, and CI that runs on my own cluster

Thu Sep 24 2026

#self-hosting #cicd #kubernetes #forgejo #woodpecker

My previous post was a tour of my Kubernetes homelab, and one section got a passing mention. The cluster hosts its own git host and its own CI, and I gave that a few paragraphs at the bottom before moving on to the service roll call. The build system deserves more than a grace note, because it’s the piece I’d defend hardest. If your code history and your deploys both live inside someone else’s infrastructure, you’ve outsourced the one thing that’s hardest to rebuild. This post is about taking it back.

The year of trusting GitHub less

GitHub’s 2026 makes the argument by itself, so I’ll keep this short.

In February, two related incidents broke pushes, pulls, Actions runs, and Copilot for about two and a half hours. The cause read like a soap opera: a config change to a caching mechanism triggered a wave of simultaneous cache rewrites, which cascaded into connection exhaustion on the git HTTPS proxy. Twice, because the fix missed a second source of cache updates.

Then August arrived. On the 6th, a routine deployment reduced pod capacity in one datacenter, the service mesh saturated, and Actions fell over for about nine hours of impact, with the published incident window running past ten. Copilot’s coding agent, code review, Pages builds, and Dependabot went with it. Eleven days later came the big one: just under eight hours of degraded everything, with error rates around twenty percent across the web experience and half of repository archive downloads failing. The postmortem traced it to a service-mesh sidecar that failed to autoscale, which saturated the load balancers in one datacenter, made worse, GitHub said, by a latent retry bug in VS Code that amplified traffic to the Copilot token service by about ten times. Twenty-nine thousand organisations saw errors across 4.8 million requests.

None of this was a DDoS or an exotic hack. It was capacity planning losing a race. The company’s own CTO said plainly that monthly commits had grown from 1.4 billion to 2.9 billion since April and that the failures were capacity problems, not bad code. Four of the five incidents that month traced back to the same kind of losing. The fifth was an upstream dependency falling over.

Every cloud service has outages, that part is fair. What changed for me is what GitHub’s outages take down when they happen. It isn’t a dashboard you like. It’s pushes, pulls, authentication, Actions, and Pages, all at once, and for the duration of the incident the software industry’s shared source of truth is unavailable. Handing over your git history and your deploys was always a bet. 2026 was the year the bet cost people days of work.

Owning the source

So I want the primary copy of everything to be mine, and I want it on hardware I can restart. That means a self-hosted Git server, and my pick is Forgejo, running on the Titan cluster with its state on a Longhorn volume and its container registry on the NAS.

Why Forgejo over the alternatives, quickly. It’s a community fork of Gitea, run by a nonprofit, and there’s no company rewriting the terms of service or deciding that features the community paid for now need a subscription. It runs happily on ARM64 in a few hundred megabytes of RAM, which matters on a Pi cluster where GitLab’s minimum footprint would eat a node by itself. It has issues, pull requests, Actions-compatible CI, and a built-in OCI container registry, which is everything I actually use. Images built on the cluster get pushed to that registry over the LAN, and other namespaces pull from it with a pull secret. Built on the cluster, pulled on the cluster, and none of it leaves my four walls.

GitLab I still run into, but as a mirror, not a home. It’s an alright replacement for GitHub on the surface: self-managed, mature CI, decent UI. Dig a level down and the future isn’t cemented. It’s a public company that ran a $57 million GAAP operating loss last quarter despite being profitable on the non-GAAP measures it would rather you quote, and it cut about 14% of its workforce in 2026, exiting 22 countries on the way, with the restructuring framed around agentic AI workloads. Its pitch is now “intelligent orchestration” for the AI era. Its licensing moves over the years have taught people what an open-core carve-up looks like. I don’t want my git hosting to have a stock ticker attached to it. Forgejo’s fate is decided by its contributors, not a quarterly call.

The mirror is a pipeline step

That’s not to say Forgejo lives in isolation. The setup I settled on covers the case where my own setup fails, and it costs less trust than it sounds.

A chunk of repos on Forgejo replicate out to a personal GitLab instance. The second copy is just a push mirror, and for the blog it lives directly in CI as one step of the Woodpecker pipeline. The primary is mine, the backup copy is a git push --all with a token, and it runs after the build and tests pass. If Forgejo and the cluster were both somehow vaporised tomorrow, every commit of the mirrored repos is sitting on GitLab’s free tier, and restoration is an import, not a dig through backups.

You can point the same firehose anywhere: GitHub, Bitbucket, a bare repo on the NAS. I deliberately keep it to one extra host, because the mirror’s job is to survive me losing the first copy, not to turn into a second primary I have to keep in sync in both directions. One primary, one cold copy.

If you want backups or fallbacks, you do not need a managed service for your source. You need a CI step that pushes somewhere else.

Runners: Forgejo’s version, and then the native one

Forgejo Actions gives you CI straight out of the box, runners included. They work. What they don’t do, by themselves, is behave the way a Kubernetes workload should. The runner is a process that sits there waiting for jobs, which on a 32GB cluster is a small insult, and the runner model boils down to “a container that wants to be a VM”. Nothing about it is k8s native.

I fixed that myself once, because it was fun. A small exporter polled the Forgejo API every thirty seconds for pending runs and published the count as a Prometheus metric. KEDA watched that metric, and when work appeared it launched an ephemeral runner pod through a ScaledJob. Zero runners when idle, one when a workflow needed it, the whole thing wired together by a burst of YAML I’m still a bit fond of.

Short version of what happened next: that’s not a hack anyone needs to write anymore. Being early has one perfect ending, and this was it: the ecosystem shipped what I had glued together, and my copy of the glue became deletable. KEDA added a native Forgejo scaler in v2.18 which talks to the Forgejo API directly and scales on the number of pending jobs, and the runner binaries gained a one-job mode so each spawned pod registers, runs a single workflow, and exits. Forgejo v15 tightened the security story further, so a runner can register itself as ephemeral and have its credentials deleted when the job is done. The scale-to-zero setup I built by hand is now about twenty lines of YAML on top of stock components, which is the correct amount. Mine is still running until I get round to swapping it, but it’s officially redundant, and the upgrade README for the next person is shorter than my exporter ever was.

So Forgejo runners scale to zero, and they work. They stayed the right answer for lightweight stuff and I keep a set of them deployed. But the runner model still has a floor: a runner pod executes the workflow’s steps inside itself, on its own terms. For anything heavier than “echo hello” you end up managing Docker-in-Docker inside the runner container or leaning on host mounts. When I compared notes with the way CI should behave on a cluster, there was a ceiling I could see from here.

Woodpecker: the steps are pods

The ceiling was where I stopped doing this by hand and switched to Woodpecker.

Woodpecker is a CI engine that grew out of Drone, released under Apache 2.0 and governed by its community rather than a company. Codeberg runs its main CI on it, which says something about scale and staying power. Features were not what sold me. What sold me is the Kubernetes backend, where a pipeline step is not a shell escape or a nest inside a runner. A pipeline step is a pod.

The version I run is a single server pod and a single agent pod, and the server claims around 100MB and the agent about 30MB at idle. SQLite in a PVC, plenty for one shop. But the shape of the thing is what’s interesting. Every step, and every service attached to a step, becomes its own pod on the cluster, with its own image and its own resource fields. Which means the pipeline file stops being a document and becomes a set of Kubernetes scheduling decisions, and you get to state them one at a time instead of betting on a pile of runner assumptions.

Here’s a cut-down version of the blog’s real pipeline showing the two ends of the scaling story:

steps:
  - name: sync
    image: alpine/git:2.49.1
    commands:
      - git push --all https://gitlab.com/<your-gitlab-user>/blog.git
    backend_options:
      kubernetes:
        resources:
          requests:
            cpu: 50m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 128Mi

  - name: test-cross-browser
    image: mcr.microsoft.com/playwright:v1.62.1-noble
    commands:
      - npx playwright test --project=firefox --project=webkit --grep "@browsers" --workers=2
    backend_options:
      kubernetes:
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            cpu: 2000m
            memory: 3Gi

Those two steps sit in the same file and they run on the same cluster. The sync step that mirrors the repo costs 64 megs of memory. The cross-browser Playwright step asks for 500m of CPU up front and tops out at 2 gigabytes, because three browser engines is real work. Between them sit the build step and the deploy step with their own numbers. Nobody size-configured a runner host to be able to do the biggest job someone might hand it. Each step gets the resources it needs, assuming your nodes can scale up or across to supply them.

The step-is-a-pod model implies more:

A step that needs Docker gets Docker-in-Docker as a sibling service pod, not layered inside the build container, which buys you straightforward TLS and skips a whole genre of overlay filesystem suffering.
Every step can pick its own image, and nothing persists between them. If the workspace needs to survive from step to step, it behaves exactly like any other k8s volume, and Longhorn supplies the RWX volume that makes this work across nodes. There’s no runner home directory to tidy.
The server and agent pods themselves are tiny, and the official autoscaler project exists to make agents themselves scale to zero if you’d rather not keep one around.

A Woodpecker run is not a contract with a runner. It’s a set of pods the Kubernetes scheduler places wherever there’s room, reschedules when one dies, and meters per pod. Small commands, deploys, and syncs take a minimum footprint. Large builds reserve what they need. This is what “k8s native” is worth paying the extra indirection for: the same old tool that schedules your stateful apps also schedules your CI, and you never have to think about which machine a job runs on, or whether the runner has RAM to spare.

The proof has a deploy URL

The blog is a Forgejo repo. You push to main, Woodpecker gets a webhook, and a pipeline runs through lint checks, an Astro build, a Chromium test pass, a Firefox and WebKit pass, then the wrangler upload to Cloudflare Pages and a push mirror out to GitLab. Every one of those steps is a pod on the cluster this post describes.

The previous post ended on a joke: the post about the cluster would go up through the cluster, and the snake finishing its tail would close the loop. That loop is closed. This one was built the same way: a push to Forgejo, Woodpecker building and testing it as pods on Titan, wrangler shipping the result to Cloudflare, and a second copy of the repo riding one pipeline step out to GitLab. The mirror that backs everything up is one pipeline step away, not one outage away.

Worth taking with you

Own the remote, not the ecosystem. Forgejo answers to its contributors, and on the day its governance ever goes sour, git is a file format and git push travels anywhere.
Backups live where your data can’t follow the disaster. A second copy pushed to someone else’s cloud is the escape hatch I never want to use, and it costs one pipeline step.
Runners should not be pets. Scale them from zero, kill them after the job, and let the scheduler and the Forgejo API do the allocation.
If you’re running CI on Kubernetes, pick CI that has a Kubernetes backend. When the steps are pods, your pipeline and your infrastructure speak the same language of scheduling, and the runner stops being the whole world.
“K8s native” doesn’t mean runs-in-a-container. It means every unit of work is a schedulable object with requests, limits, and a lifecycle that Kubernetes owns. That difference is worth more than any feature list can sell you.

Forgejo holds the sources, Woodpecker plans the builds, and the backups sit one pipeline step behind them. My code lives somewhere its future isn’t someone else’s roadmap, and the machine that builds it is three Pis that I can power cycle without having to email anyone. There are nicer platforms, and there are larger ones. I measure mine in hours spent watching someone else’s status page, and that number is zero. Better is when it’s yours.