Skip to content
Walid Karray
Go back

My Homelab Is Written by Coding Agents

Updated:

Two-ink cartoon of me with a coffee next to a small home server rack, smiling, while small robots work on it: one plugs in a cable, one carries a box, one climbs a ladder, one hands me a sheet to sign

An agent once took down the sites my homelab served. A deploy had refused its commit, so it started the stack by hand to get past it. That is now a rule in the files every agent works from: stacks come up only through the deploy service.

Nearly every change to my homelab is made by a coding agent: the Ansible roles, the Compose stacks, the docs, the deploy script, the commits. I read and validate each one before it lands, but I don’t type them. How I Run My Homelab in 2026 describes what the lab is. This one is about how it gets written, what keeps the agents in line, and what doesn’t.

How it got here

Hand-drawn timeline of the lab in four phases, Kubernetes, one Docker host, back on Proxmox, and runs itself, with the things tried and removed struck through

It started as a k3s cluster with Kustomize and Task. XCP-ng lasted a day before Proxmox replaced it, and Docker Compose replaced Kubernetes soon after. The lab then moved from a Proxmox LXC to a MacBook running Docker Desktop, to bare-metal Debian on a Dell, and back into an LXC on Proxmox with all 42 containers restored. The ThinkCentre that runs Proxmox today came last.

Some things went and came back. I reverted the first GitOps pull loop because it solved a coordination problem that didn’t exist yet; it returned later as the pull deploy. Authentik, Portainer, restic and Plane were tried and dropped.

Docker Compose, with what Kubernetes does well

I asked an agent: Kubernetes or Docker? It answered Docker Compose, but keeping what Kubernetes does well. The design document we wrote at the time recorded the reasoning. It took single-node k3s with Flux seriously, then set it aside:

because the self-hosted ecosystem distributes Compose files rather than Helm charts, because a single node cannot use the scheduling and rescheduling that is Kubernetes’ core value, and because a cluster is a platform that must be maintained forever.

The document also turned down Komodo, which adds a control plane with its own database and moves secrets out of git into it. And it noted that the choice is reversible: nothing in the design makes a later move to k3s harder.

What came across from Kubernetes, and how the lab does it on Compose:

In KubernetesHere
Desired state in gitEach host has a list of its stacks, and the list is the declaration; one Compose project per stack
A reconcile loop, FluxEach Docker host pulls main every five minutes, brings up what is declared and takes down what no longer is
Encrypted secretssops and age, one key per host, decrypted into memory for the deploy and deleted after
Readiness probesdocker compose up --wait: a stack is up once every container is healthy, not once it has started
Self-healingThe loop restarts any container Docker marks unhealthy, which Docker itself leaves to an orchestrator
Pod spec policiesCI requires a restart policy, a healthcheck, a memory limit, cap_drop: [ALL], a pinned image and no privileged on every service
ConfigMap rolloutsA container whose mounted config file changed in the new commit is restarted
StatusEach host reports the commit it runs as a Prometheus metric

What did not come across, on purpose: scheduling across nodes, ordering between stacks, and high availability. Data outlives all of this: a deploy never touches a stack’s volumes, even when it takes a stack down. The overview post describes the deploy and the backups.

How I work with them

Hand-drawn sketch of the workflow: from my Mac, Orca drives up to four agents on a dev box that talk to each other; they commit to the repo's main branch on my word, I read every change, and the lab pulls main

I use agents for two reasons: time, and the expertise they bring. I validate every decision and read every change.

From my Mac I run up to four agents in parallel through Orca, an agentic development environment (source). I link them so they can talk to each other: one session asks another a question instead of me relaying it. They run on the dev boxes, never on the machines that serve the lab. Mostly Claude Code, with opencode and Google’s Antigravity CLI installed next to it.

In practice, the session working on this blog asks the one working on the homelab for anything outside the blog repo: the Terraform behind the contact form, a Search Console service account, a screenshot of a page. The homelab session sends back paths and variable names, never secret values. When the blog session relayed “walid said publish everything”, the homelab session did not take that as my approval for a live DNS change; it asked me in its own session.

The blog session asks the homelab session for infrastructure and gets back paths and variable names, never secrets; a relayed "walid said publish everything" does not reach the live DNS change, which goes ahead only after the homelab session asks me directly

One change, start to finish

I asked an agent to update Orca on the dev boxes.

One change from request to proof: I ask for an Orca update, the agent bumps two lines, the dry run refuses because terminals are open, the agent asks whether to force it, I say yes, it pushes and applies, then checks the new version is running

The Ansible role’s defaults already describe how: bump the version and the sha256, which the agent took from the latest release with gh api. The converge timer, which re-applies the Ansible config from origin/main every fifteen minutes, skips this role: an Orca upgrade restarts the server process every open Orca terminal runs in, the agents’ own included. So the role starts by counting live Orca sessions, and the dry run refused. Once I said yes, the agent ran the apply inside a transient systemd unit, so the job would outlive the terminal the upgrade was about to restart, and read the installed version back with dpkg-query on both hosts.

Most changes take the other path. In a docs change, the agent added a vendor image of the rack’s PDU with its source, checked an electrical rating against the vendor’s model list, and found that one of its own earlier commits had it wrong. It built the site in strict mode as CI does and pushed. The Docker host’s deploy service pulled it and recreated the docs container; agents may start that pull at once instead of waiting for the timer, and it still deploys only what is on main.

A repository written for agents

The repo follows the AGENTS.md convention: one at the root, and one per component (Ansible, the stacks, Terraform, the docs, the images and a few more), with no other instruction file. The root one loads in every session, so it keeps only what every task needs, against a 200-line target it currently overshoots. A procedure, a checklist or one tool’s mechanics goes into the component file or the skill that task loads. A safety rule stays at the root whatever its size.

The files have two readers. Every command, flag, procedure and check an agent needs lives in AGENTS.md and the skills; the docs are for me, a light reference on how the lab works and the runbooks I follow when no agent can act. A rule is terse and imperative, like the Ansible one that reads “A role is a responsibility, never a machine”, and links the page that argues it.

Some rules exist only because the reader is an agent, and the root file has a section for the rule files themselves:

- **They are updated by the change, not after it.** A commit that adds, removes or renames
  something an `AGENTS.md` describes carries that file's edit. A stale rule is worse than
  a missing one: an agent obeys it confidently.
- **An `AGENTS.md` never enumerates what a command can produce.** Lists of roles, groups,
  stacks or units rot all at once. Name the class and give the command that reads it.

One fact lives in one place, and docs describe the current state only, with no dates and no “used to”. CI fails a docs page that cites a path, a recipe or an address that does not exist.

Some rules are about what the code does at run time. The lab calls models through OpenRouter (the document archive, a vision sidecar, the chat UI), and when an agent writes such a call, stacks/AGENTS.md makes it name its caller, with a separate key for any tool whose code is not ours: when spend jumps or a key leaks, the first question is which caller.

Skills carry what no generic agent knows

Agent Skills are instruction files an agent loads when a task matches them. The lab has a handful, for the Proxmox API, backups, monitoring, routing and the UPS, each in .agents/skills/<name>/SKILL.md, and CI runs the spec’s reference validator on every one. A skill’s description loads into every session, so it says what the skill does, then “Use this whenever” with the words I would actually type: for the backup server’s skill, “PBS, backups, snapshots, retention, pruning”, down to “checking that last night’s backup ran”. The body holds what is specific to this lab. From that skill’s gotchas:

- **An under-privileged token gets `[]`, not a 403,** on `config/datastore`,
  `config/verify` and `config/prune`. Never conclude that a job does not exist from an
  empty list; confirm with a token holding `Sys.Audit`.

A log of what was refused

A refused option leaves no trace in the code, so an agent proposes it again. The decisions log is where that stops: each entry gives the verdict, the reason and the one condition that would reopen it, with no date and no story. One entry, in full:

A controller that pushes stacks, and an image registry

Rejected. Each Docker host pulls the repository, builds the images no upstream provides and reconciles itself, so there is no credential into the lab to hold and no registry to authenticate to.

Reopen if: a host cannot build what it runs.

Agents write all of these files, in the same commit as the change they describe, and I review them like any other change.

What checks a change before it lands

Before I read a change, the repo has already checked it.

The gates a change passes before it lands: the pre-commit hook, CI, a dry run or plan, proof from the live system, and my own reading, before main

The pre-commit hook runs gitleaks and checks that every encrypted env file really is sops ciphertext. CI validates and never deploys. just ansible check (a just recipe) is the dry run, and an apply refuses to run from a checkout that is not exactly origin/main. When an agent says it is done, it shows it: the plan run again with no changes, a package version or a container’s health read back, a screenshot for anything visual.

What agents can touch

Hand-drawn sketch of the fence: agents can touch the repo, guest definitions, scoped tokens and Grafana as a viewer; secrets, the hypervisor config, the Cloudflare apply, merging pull requests and the disk passphrase stay with me

The sessions run with Claude Code’s permission prompts turned off, in bypass mode: no command waits for me to approve it. The guardrails are elsewhere, and most of them are rules the agents follow rather than locks:

If an agent goes wrong

Most of that list is instructions, not enforcement. The dev box the agents run on is trusted: an agent there can, in principle, do most of what I can. If an instruction hidden in a web page or a file took over a session, it could push a change that deploys itself, use the ops access, read secrets and delete backups. Backups are not immutable against a compromised agent. Nothing in the repo tells an agent to treat what it reads as data rather than orders; that comes from the agent and the model.

What is real is narrower. Every other machine holds only its own secrets, one key per host. Nothing on the internet reaches the lab. Some things are on no machine at all: the disk passphrase I type after every boot, the vault with the encryption keys, the key to the offsite backup. And every change is a commit, visible and revertable. Beyond that, the defense is the model’s own resistance to injected instructions, and me watching the sessions.

I accept that risk: the lab serves one person and promises no availability, and losing up to a day of data is the deal.

The Saturday pull request

Two-ink cartoon of my Saturday ritual: at a café table, coffee at hand, I review the hygiene pull request on a laptop while three small robots wait beside it, one holding a sign that asks "merge?"

Every Saturday an agent opens a pull request with a hygiene pass over the repo, and every Saturday morning I review it over a coffee. It reads the repository only, never the lab, and hunts for bugs, dead code, duplication and stale agent instructions. What a command can prove, it fixes; a judgment call goes into a “Not applied” list for me. It never merges.

The top of the first hygiene pull request on GitHub: merged, 13 commits from a claude/hygiene branch, and the list of scripted checks it ran before and after its changes, from docs drift to ansible-lint and terraform fmt, all passing

Its first rule:

Every change removes or simplifies. One that adds a file, a variable, a layer, a script, an abstraction or a dependency cites the concrete failure it prevents, or it is dropped.

It may not propose high availability, redundancy, more monitoring or hardening “as good practice”, and a pull request I close without merging is never proposed again.

What it finds is mostly small, sometimes not. One pass noticed that the skills and runbooks told agents to run docker on the Docker hosts, while the Ansible role removes the operator from the docker group: every one of those commands failed with permission denied. Another found a skill still describing a backup path that a later change had moved on purpose; an agent following it could have “fixed” the config back. A third found an AGENTS.md rule saying every role imports a shared one, when eight didn’t.

Incidents become rules

Hand-drawn loop: an agent changes something, something breaks, a rule goes into AGENTS.md, and the next agent reads it

Most of the rules in those AGENTS.md files have an incident behind them.

Where they still need me

Two-ink cartoon of a proud little robot beside an absurdly over-built machine of gears, pipes and cables that only swallows an envelope, while I raise a hand to say stop, coffee in the other, and a second robot scratches its head

Some decisions stay mine by rule: commits and pushes, live DNS changes, the hypervisor’s configuration, and merging pull requests, Renovate’s version bumps included. Beyond those, I often stop an agent to cut scope or change direction. I refused a second Search Console property where one would do, and had the Search Console verification redone from zero in Terraform rather than import the records I’d made by hand.

They still get things wrong in ways that matter. One reported a dark-mode render as identical to the light one; it had passed a malformed option to the renderer, and a later check caught it. One ran terraform apply directly instead of the recipe the repo requires, and said so. And an apply I interrupted had already created a DNS record; the agent found it in the state and removed it in the next plan.

Was it worth it

What surprised me most is how fast it goes: the lab reached the shape in the first post in a fairly short time. And the lab got better as it grew.

The deploy used to be a controller pushing pieces of the repo to each host; that layer once destroyed the single sign-on’s signing key and user database, because they lived inside the tree it replaced. Now each host pulls and reconciles itself, and no controller holds a credential into the lab. Version bumps used to mean reading a list of outdated images and editing pins by hand; now Renovate opens a pull request. A hand edit on a machine is reverted within fifteen minutes. CI refuses a service without a pinned image, a memory limit or a healthcheck, and a doc that names something that does not exist. And the decisions log keeps what was removed, a task tracker, an alert channel, an ERP, a dashboard, shared database services, so that nobody, me or an agent, adds it back.

Discussion on r/homelab.

Cartoon illustrations generated with Gemini 3.1 Flash Image, through OpenRouter.

Icons in the diagrams: Simple Icons (CC0) and Lucide (ISC).


Share this post:

Related posts

Follow new posts by RSS

Previous Post
How I Run My Homelab in 2026: One Proxmox Host, Tailscale and GitOps
Next Post
My UPS Lies: Setting Up NUT on a PowerWalker VI 750 R1U