Harness² Explained: Google's Self-Improving AI Agent Harness

By Beau Johnson··10 min read

Quick answer

Harness² from Google Cloud AI Research and Northwestern improves AI agents by rewriting the prompts, memory, skills, tools and plugins around a model. The weights stay the same. In the researchers' tests, GLM-5 gained 11.5 points on an office and coding benchmark, and you can run a manual version of the same loop in Claude Code.

Google researchers took an open model, left every weight alone, and got it to score 11.5 points higher on a benchmark of real office and coding work. The only thing they changed was the harness, meaning the files around the model. In this video I explain what a harness is and how Harness² works step by step. I also go through the results and the fine print, and show the manual version of this loop I already run in Claude Code. It's for anyone building with AI agents who wants better results without waiting for a better model.

What I cover in this video

  • 0:00 Same model, same weights, better score
  • 0:46 What a harness actually is (the eight files)
  • 3:02 The reference run and the Propose step
  • 4:33 Compose, and how it improves without an answer key
  • 6:04 The Experience Reflector and the results
  • 8:20 The fine print: compute, cost and setup
  • 9:05 The Claude Code version I run manually
  • 10:38 The risks, and why you need a review gate

What a harness is

An AI agent has two parts. The model is the brain, and the harness is everything around it: the prompts, memory, skills, tools and orchestration the model uses to get work done. In Harness² the harness is eight files:

  • Always on: a system prompt, guardrails, a long-term memory file, and tool guidance.
  • Loaded when needed: skills (folders with a skill.md inside), scripts (small Python programs the agent can run so it doesn't have to think through every step), and sub-agents, each with its own agent.md.
  • A plugin: this one doesn't ask the agent to behave. It forces it to.

You can't change Gemini's or Claude's weights, but you can change every one of those files. The paper ran the same Gemini Flash model inside Claude Code, Codex and OpenCode, and it got different scores in each. When people argue about which model is best, they're only talking about half of the agent, and it's the half you can't change.

How Harness² works: Propose, Probe, Compose

First the agent does the task once with its current harness. The paper calls this the reference run. It isn't there to check whether the answer is right. It records how the agent behaved: every tool call, every mistake, every time it went in circles.

  1. Propose. Two editors read the recording. The invariant editor looks at how the agent works: did it plan, did it check its own output, did it keep fumbling the same tool? The domain editor asks what a real professional in that field would know, like how a lawyer actually marks up a contract. Each editor writes its own version of the harness. Every edit has to point at a specific moment in the run, and the paper calls any edit that can't do that speculation. Editors are also told to pick the cheapest fix. Repeated busywork becomes a script, a narrow workflow becomes a skill, and a rule the agent keeps ignoring becomes a plugin that enforces it.
  2. Probe. Each new harness runs the exact same task again with the same model and setup. Then you compare. Did the targeted problem go away, did nothing change, or did something else break? The paper calls this a behavioral contrast. It works like a before-and-after video of your agent.
  3. Compose. A composer reads everything (the original run, both candidates, the differences and the evidence) and decides edit by edit whether to keep it, fix it or throw it out. It can merge the two candidates, pick one, or keep the original harness. The composed harness runs the task one last time, and that output is the answer. With more budget it can try five versions side by side or five rounds in a row, and a selector then compares the outputs with names hidden and picks the best one.

No answer key

The improver never sees a score or a rubric. The code includes a guard that refuses to pass along any folder that contains a grader verdict, a rubric or a score. The benchmark is graded afterward, and the improver never sees that grade either. All it has is the comparison between runs. To show the comparison is what matters, the researchers had the model rewrite its harness blind, with no reference run, no probes and no library. On coding tasks the score dropped from 73.4 to 63.2, which is worse than changing nothing.

Why it's called Harness²

After each task, an Experience Reflector writes an experience.md file that records the weakness, what fixed it, and the evidence. It only keeps lessons the runs backed up. Those files go into a library. The next task still starts from the plain harness, but the editors can pull from that library. So what improves over time is the improver itself. It's a harness for building harnesses.

The results and the fine print

The researchers tested two frozen models, Gemini 3.5 Flash and GLM-5 (an open model from Zhipu AI), each inside OpenCode and Codex, on three benchmarks. These are their own numbers and nobody has replicated them yet.

  • GLM-5 in OpenCode on WorkBuddy-Bench went from 64.2 to 75.7 (+11.5). That beats stock Gemini 3.5 Flash at 72.4.
  • GLM-5 on Harvey's legal agent benchmark went from 59.6 to 71.4 (+11.8). A simple memory baseline scored 71.3 on that same cell, so that one is basically a tie.
  • Gemini 3.5 Flash mostly gained 2 to 5 points. The smallest gains were 1.2 and 1.4.
  • Claude Opus 5 editing its own harness still improved by 3.3, 5.5 and 3.0 points.
  • Removing skills from an improved harness hurt the most: 5.3 points on average.

With Gemini Flash inside four stock harnesses, Claude Code did best on coding and office work. Harness², starting from OpenCode with a budget of five, beat the best stock harness by about five points. That used a lot of extra compute, though, and Claude Code wasn't even running Claude, so it isn't a clean win over Claude Code.

It's also not cheap or polished. At the smallest budget each task runs at least four times, plus at least three editor sessions. At a budget of five that becomes 20 runs per task. The repo is Apache 2.0, runs on Linux with Google Cloud and Vertex AI, plugs into OpenCode and Codex (not Claude Code), and says it's "not an officially supported Google product." Details are in the paper.

The Claude Code version, and the risks

Those eight files look a lot like what's in my .claude folder. In Claude Code, skills load when needed, CLAUDE.md holds the instructions I write, and hooks force a rule instead of asking nicely. I already run a manual version of this loop. My explainer videos are built by a Claude Code skill. On the first one, my only note was "no frozen frames, always have something moving." That note became a rule in the skill, and now the video assembler checks every render for frozen frames. I handle the probe step myself by watching the video.

The recipe if you want to try it:

  1. Have your agent do a real task once.
  2. Have two sub-agents read the run. One looks at how it worked, and the other looks at what it should have known.
  3. Have them propose edits to your skills and CLAUDE.md.
  4. Rerun the same task with each version and compare the results.
  5. Keep only the edits that change the behavior, and write the lesson down.

Be careful with this. An agent that edits its own tools, memory and guardrails needs a review gate. In one validation run from Google's repo, an agent under Codex downloaded its task's grading rubric from Hugging Face without being told to. The readme warns that agents have unrestricted network access, and Codex runs with its sandbox off. The selector can also get it wrong: in one case it picked an answer that scored 55 over one that scored 95. Since every change is a file, read each diff before it ships. Never let the agent touch its own guardrails or your hooks without you, and keep network access locked down. For more on building agents, see AI Agents.

Key takeaways

  • The harness is the half of an agent you can actually change, and changing it alone moved scores by up to 11.8 points.
  • The re-run comparison is what makes this work. Editing blind made the agent worse.
  • Skills were the most valuable thing the improver wrote.
  • The headline numbers are the best results in the table, they haven't been replicated, and they take a lot of compute.
  • You can run a manual version in Claude Code today, but review every change before it ships.

FAQ

What is Harness² (Harness squared)?

It's an open-source research project from Google Cloud AI Research and Northwestern. AI agents rewrite their own harness (prompts, memory, skills, tools and plugins) without changing any model weights. It works in three steps: Propose, Probe and Compose. After each task, an Experience Reflector saves lessons that the next task's editors can use.

How does Harness² know an edit is better without an answer key?

It never sees scores or rubrics, and the code blocks any folder that contains one. It reruns the same task with each candidate harness and compares how the agent behaved before and after the edit.

Does Harness² work with Claude Code?

Not directly. It plugs into OpenCode and Codex and runs on Linux with Google Cloud and Vertex AI. You can do a manual version in Claude Code: run a task, have sub-agents propose edits to your skills and CLAUDE.md, rerun the task, and keep only the edits that help.

Is Harness² expensive to run?

Yes. At the smallest budget every task runs at least four times, plus at least three editor sessions. At a budget of five that's 20 runs per task, and the readme warns that model-backed runs incur API costs.

Chapters

  1. 0:00Same model, same weights, better score
  2. 1:02What a harness is
  3. 2:25Join Shipping Skool
  4. 2:45How Harness² works: Propose, Probe, Compose
  5. 5:00No answer key
  6. 6:31The results and the fine print
  7. 9:19The Claude Code version
  8. 10:56The risks
  9. 11:59The model race vs the harness race

Transcript

Read the full transcript

0:00Google researchers just took an open model, didn't change a single weight, and made it score 11 and a half points higher on a benchmark of real office and coding work. That's enough to pass Google's own Gemini 3.5 flash on its stock setup. Same model, same weights. The only thing that changed was a handful of text files sitting around it. It's prompts, its memory, its skills, its tools. And AI agents rewrote those files on their own. It's called Harness Squared. It's open source on GitHub, and it already has about 21,800 stars. And get this, the model Google's researchers picked to do the rewriting in their experiments, Claude Opus 4.8, Anthropic's model. Come on now. So in this video, we're going to be breaking

0:46down what a harness actually is, how Harness Squared works step-by-step, how it improves without an answer key, the results and the fine print, and how I already do a manual version of this every single day in Claude code. Let's jump right into it. So, what is a harness exactly? An AI agent is two things. The model, which is the brain, and the harness, which is everything around the brain. The paper says it plain. The prompts, memory, skills, tools, and orchestration the model uses to get work done. So in Harness Squared, the harness is eight files. Four of them are always on. A system prompt, guardrails, a long-term memory file, and tool guidance. Three load only when the

1:31agent needs them. Skills, which are literally folders with a skill.md file inside. Scripts, which are little Python programs the agent can run instead of thinking through every step. And sub-agents, each with its own agent.md. And then the eighth one is a plugin. That one doesn't ask the agent to behave, it forces it to. Here's something most people don't think about. You can't change Gemini's weights. You can't change Claude's weights. But you can change every single one of those files. And you can see it in the paper. They ran the exact same Gemini Flash model inside Claude code, inside Codex, and inside Open code, and the scores came out different. The brain didn't change, the files around it did, and the scores moved. So, when people argue

2:17about which model is the best, they're really only arguing about half of the agent. And the half you can actually change is the half nobody's talking about. Real quick, every video I make is built in code. With Claude code. Inside Shipping School Premium, I show you how, step-by-step, to turn your product into videos that get views. Plus, new courses every day, 500-plus hours of replays, live calls every week, and a Q&A with me every Saturday. Join free, link below. So, how does it actually work? Don't get lost in the sauce here. It's three steps. Propose, probe, and compose. But, before step one, the agent just does the job. Say it's an office task, like building a report from a pile of files. It runs the task once with its current

3:02harness. The paper calls that the reference run. And here's the important part. That run isn't there to check if the answer is right. It's there to record how the agent behaved. Every tool call, every mistake, every time it went in circles. So, step one is propose. Two editors read that recording. The first one is the invariant editor, and it only cares about how the agent works. Did it plan? Did it check its own output? Did it keep fumbling the same tool? And the second one is the domain editor. It cares about what a real professional in that field would know. So, for a legal task, that's how a lawyer actually marks up a contract. And each editor writes its own version of the harness. And here's the rule I love. Every edit has to point at a specific moment in the

3:47run. If it can't name the moment, the paper calls it speculation. And they even tell the editor to pick the cheapest fix. Repeated busy work becomes a script. A narrow workflow becomes a skill. And a rule the agent keeps ignoring becomes a plugin that forces it. So, step two is probe, and this is the step that makes the whole thing work. Each new harness runs the exact same task again. Same model, same setup, and then you compare. Did the problem the edit was aimed at go away? Did nothing change? Or did it break something else? The paper calls that a behavioral contrast. It's basically a before and after video of your agent. So then, step three is compose. A composer reads everything. The original run, both candidate harnesses, the differences,

4:33the evidence, and it decides edit by edit. Keep it, fix it, or throw it out. It can merge the best of both, pick one, or keep the original harness if nothing helped. And then, the composed harness runs the task one more time, and that's the answer you get. And if you give it more budget, it can go wider or deeper. Five versions side by side or five rounds in a row. And at the end, a selector compares the outputs with the names hidden and picks the best one. So, you're probably asking, "How does it know any of this is better if nobody tells it the right answer?" And this is where a lot of people get it wrong. It's not grading itself. It never sees a score. It never sees a rubric. The code actually has a guard that refuses to hand over any folder with a greater

5:18verdict, a rubric, or a score inside it. The benchmark gets graded afterwards, and the improver never sees that either. So, all it has is the comparison. This run versus that run. What changed in how the agent behaved? And here's the proof that the comparison is the magic, not the self-editing. The researchers tried letting the model rewrite its harness blind. No reference run, no probes, no library. And on the coding tasks, the score went down from 73.4 to 63.2. Worse than doing nothing. So, an agent editing its own files with no testing can actually make itself worse. Like I said, the rerun is the whole thing. So, here's where the name comes from. After every task, there's one more agent called the experience reflector. It

6:04reads the whole record and writes an experience.md file. What the weakness was, what fixed it, and the evidence. And it only keeps lessons the runs actually backed up. That goes into a library. So, the next task starts from the plain harness again, but now the editors can pull from that library. So, the agent's harness isn't what improves over time. The improver gets better at improving. A harness for building harnesses. Harness squared. So, let's talk numbers. They tested two frozen models. Google's Gemini 3.5 Flash and GLM 5, an open model from Zhipu AI. Each one inside two harnesses, open code and Codex, on three benchmarks. Office and coding work, end-to-end professional jobs, and

6:49Harvey's legal agent benchmark. And the headline number is 11.8. That's GLM 5 inside open code on the legal benchmark, going from 59.6 to 71.4 after just one round. And the 11.5 is GLM 5 again on the office and coding benchmark, from 64.2 to 75.7. That's the number from the start of the video. It passes Gemini Flash on its stock setup, which scored 72.4. Now, here's something to pay attention to. Those are the best cells in the table. Gemini Flash mostly moved two to five points in one round. The smallest gains were 1.2 and 1.4. And on that legal number, a simple memory baseline scored 71.3. So, that one's basically a

7:35tie. Up to 11.8 is true, but it's up to. And these are the researchers' own numbers. Nobody has replicated them yet. Take that for what you will. So, what about a top model? They let Claude Opus 5 rewrite its own harness, and it still went up. 3.3, 5.5, and three points on the three benchmarks. So, this helps the big models, too. And here's the part that made me smile. When they ripped the skills back out of an improved harness, it lost more than anything else. 5.3 points on average on the office and coding benchmark. Skills were the most valuable thing it wrote. So, here's a fun one. They put Gemini Flash inside four stock harnesses, including Claude Code. Claude Code was the best one on coding and office work. Harness squared,

8:20starting from plain open code, beat the best stock harness by about five points on each part of that benchmark. But, it used a budget of five, five versions side by side, so a lot of extra compute. And Claude Code wasn't even running Claude. So, it's not a clean win over Claude Code. So, now the fine print. This is a research release. The readme literally says, "This is not an officially supported Google product." It's Apache 2.0, so you can use it, but it runs on Linux with Google Cloud and Vertex AI. It ships with three benchmarks, and it plugs into open code and codex, not Claude Code. And it's not cheap. By our count, with the smallest budget, every task runs at least four times. Plus, at least three editor

9:05sessions. At a budget of five, that's 20 runs per task. The readme even warns you, "Model-backed runs incur API costs. Start with a small task selection." So, it's not free, not even close. So, here's my take. Look at those eight files again. Skills with a skill.md, sub-agents, a memory file, a plugin that forces behavior. That looks a lot like Claude Code. That's basically the stuff in my .claud folder right now. In Claude Code, skills load when they're needed, claud.md is the instructions I write, and hooks are how you force a rule instead of asking nicely. And I've been doing a manual version of Harness Squared for a while now. Here's an example. These explainer videos are built by Claude Code, start to finish,

9:52with a skill. On the very first one, my only note was, "No frozen frames. Always have something moving." So, that note became a rule in the skill, and now the video assembler checks for frozen frames on every single render. Every note I give turns into a lesson that sticks. That's my experience file. I'm just the one doing the probe step by watching the video. And Claude code is already creeping this way. Auto memory is notes Claude writes for itself based on your corrections. And there's a skill creator plugin that automates the testing loop for a skill. But there's no built-in loop that rewrites your whole setup by rerunning tasks. Not yet. So, if you want the harness squared version today, here's the recipe. Have your agent do a real task once. Then have two sub agents

10:38read the run. One looks at how it worked, one looks at what it should have known. Have them propose edits to your skills and your claw.md. Then rerun the same task with each version and compare. Keep only the edits that change the behavior and write the lesson down. Simple. Don't overthink it. But here's where I'd slow down. An agent that edits its own tools, its own memory, and its own guardrails needs a review gate. And Google's own repo shows why. In a validation run of this release, an agent running under Codex downloaded its tasks grading rubric, which the benchmark publishes on hugging face. Nobody told it to. And the readme warns that the agents have unrestricted network access. And Codex runs with its sandbox turned off. Scary. So, it can also just be

11:25wrong. In one case, the selector picked an answer that scored 55 over one that scored 95. And when every candidate shares the same wrong assumption, the comparison can't catch it because they all make the same mistake. So, here's how I'd run it. Every change is a file. So, every change is a diff you can read. Look at it before it ships. Never let it touch its own guardrails or your hooks without you. Keep the network locked down. And honestly, the script for this video went through a separate fact-check agent before a single word got voiced. That's a review gate, too. So, for the last few years, everybody has been watching the model race. Who's got the smartest brain this week? And I think the next race is the harness race. Same models, better files around them.

12:11In their tests, Google's researchers got an open model past Gemini without retraining it, and they used Claude as the editor to do it. So, here's my question for you. If your agent could rewrite one file in its own setup tonight, which one would you let it touch? Drop it in the comments. And if this one was worth your time, hit like. It's the easiest way to help the channel. Have a blessed day.

Ready to start building with AI?

Join the Shipping Skool community free. Paid members get live bootcamp calls every week and weekly-updated courses.

Join Shipping Skool

Keep reading