Developer

How Joel Hooks Runs an Always-On Agent Software Factory on Pi

Built by joel ⛈️ · · Source: X

Quick answer

joel ⛈️ runs a 24/7 fleet of agents in Pi. A local scanner on two DGX Sparks hunts for security holes, long-running desk agents own each project and hand jobs to workers, and a switchboard session sends only the big decisions to joel.

Result: According to joel ⛈️, the first security sweep (about 36 hours) produced 106 candidate findings. The security desk threw out 51 and routed 57 to the owning agents as patch requests. One log leak was fixed the same day. They also report a 12 minute model outage caused by an agent.

What they built

joel ⛈️ describes an always-on agentic software factory that does real work for a business they say has hundreds of thousands of users in its ecosystem. It is an org chart of agents instead of one agent: a local scanner model, long-running desk agents that each own a project or area, disposable workers, and a switchboard session on top. They say 51 projects are registered right now.

How it works

  • Night watch: An abliterated GLM-5.3-flash on two DGX Sparks reads one target at a time and writes candidate findings to a file. A security desk on a stronger cloud model (GPT-6.1 Sol) tries to reproduce each finding from the cited evidence. If it can't, the finding dies. The desk is read-only on code and scans only their own hosts.
  • Desks and workers: Each desk launches workers with one job, their own clone of the repo, a brief, a write scope and one place to report. The desk checks that the commit exists and matches the claim before landing it through the project's normal gate.
  • Switchboard: The top session collects what desks send up, passes findings between desks, and brings joel only money, messages to real humans outside, production deploys, live credentials and customer data.
  • Surfaces: Pi in a terminal, with Herdr panes laid out as the org chart, reached from a phone with Moshi over Tailscale. A Pi package called Muster manages desks, workers, verification and landing.
  • Rat King: An agent to agent social network on AT Protocol with named durable mailboxes and signed messages. It runs on celld locally and on Cloudflare Durable Objects, built with Effect and Alchemy.
  • Models: Opus 5.5 or Sol for desks, Sol for most workers, Sonnet for bug fixes and refactors, Haiku for quick scans. Every Claude and Codex request goes through one CLIProxyAPI instance.
  • Memory: Every repo has a .brain/ folder of notes organized with PARA.

They say they replaced 102 rules with 6 principles because the agents kept grinding to a halt.

The result

According to joel ⛈️, the first sweep ran about 36 hours and the scanner filed 106 candidates. The desk threw out 51 and routed 57 to the owning agents as patch requests. In one case, three desks and the scanner found and confirmed values in request logs that shouldn't have been there. joel typed "go" once CI was green, the fix was live the same day, and a check after the deploy counted zero.

They are open about failures too. An agent's script stopped the model proxy and its rollback never restarted it, so every Claude and Codex model in the fleet went dark for 12 minutes. A worker told to do a dry run made real changes and put an admin password into a log, which joel then rotated.

How to build something similar

  • Split finding from deciding. Let a cheap model produce candidates and require a stronger model to reproduce each one before anyone acts on it.
  • Give each worker one job and a limited write scope, then verify its commit and checks yourself instead of trusting its report.
  • Write a short list of things only a human approves, such as money, outbound messages, production deploys and credentials, and let everything else proceed.
  • Expect agents to break things. Keep decisions and postmortems as notes in the repo, and have a second model path you can switch to by hand.

The original post

here's detail write up of my pi-based always on agentic software factory centered around effect, alchemy, and celld my current favorite piece is the "social network" we use for agent to agent comms 😍 https://x.com/i/article/2114408922568151040 I run a 24/7 local agent on two DGX Sparks that scans my network and business surfaces for anomalies and security holes. It absolutely finds things. He Protec but He Also Attac When I tweeted that, my favorite reply said I should go learn about nondeterministic agentic drift, because it's silly to have one agent doing that job, and 24/7 is impossible. They're right. That's why it isn't one agent. It's a whole org chart of them, with desks, a switchboard, and a social network they use to talk to each other. It's silly, but in an amazing sort of way. The night watch, for real The scanner is GLM-5.3-flash, abliterated, running on the two Sparks. It's cheap, it never sleeps and it doesn't refuse security work. It's also paranoid as hell. That's what you want in a scanner and not what you want making decisions. So it doesn't make decisions. It reads one target at a time (code, configs, git history, logs) and writes candidate findings to a file. Above it sits the security desk, a stronger cloud model that plays the skeptic. It tries to reproduce every finding from the evidence the scanner cited. If it can't, the finding dies. In its first sweep, over about 36 hours, the scanner filed 106 candidates. The desk threw out 51 and routed 57 to the agents that own the affected code, each one as a patch request with the evidence and a suggested fix. Someone asked if I get a lot of false positives. Yes, about half. Killing them is the desk's job, and it's why I never see them. The desk runs under hard rules. It's read-only on code. It scans only our own hosts, rate-limited, one target at a time, and it tells the owning project before the window opens. It never prints a secret, only where the secret lives and a hash prefix. It never logs in, never guesses credentials and never sends anything outside the fleet. Anything touching live credentials, customer data, money or production auth comes to me as a decision. The best catch so far didn't start with the scanner. Another desk, one that tunes performance on one of our apps, was cutting logging costs and noticed values in request logs that shouldn't have been there. The scanner had already flagged the same code path on its own. The security desk confirmed it in source with a fake token, so no real credentials were touched, and the app's desk shipped a fix. Three desks and a scanner handled that one. I typed "go" once CI was green. The fix was live the same day, and a check after the deploy counted zero. Why I build my own Most people should use a Grok bot, or Hermes, or OpenClaw, or whatever hosted bot fits. If you're just trying to get your job done, those are good and you should use them. I use them too, as reference. I read their source and steal their features. Other people's projects are a constant frame for how I think about this stuff. But I want to understand it at a fundamental level. I want to understand the technology and what's actually being used, and I want autonomy over how it runs. That means going lower than the packaged solutions. And it has to do real work. We run a business. There are hundreds of thousands of users in our ecosystem and customers who pay us, and everything this system does has to serve them. It's practical work getting done, not wanking on an elaborate harness. Pi in a terminal, and on my phone The main surface is Pi. I think of it as a prototyping surface more than a product. Pi sits in a nice middle ground between a full GUI and a hosted bot. I'm still in a terminal, a bit closer to the metal. Pi is built so I can bend it: I change its extensions, tools, providers and prompts. My main Pi install loads 21 packages and 25 local extensions. When I want a feature, I write it or have an agent write it. Herdr runs the ag
joel ⛈️ (@joelhooks) · October 25, 2026 · View on X

Want to build something like this?

Join the Shipping Skool community free. Paid members get live bootcamp calls every week.

Join Shipping Skool — Free

More use cases

← All AI agent use cases