HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Paper: HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? (v1) https://arxiv.org/html/2609.01437v1

HarnessDev: build the harness, then test the revision

A reliable stopping rule belongs to agent infrastructure, not to the wording of a confident final answer. HarnessDev evaluates that infrastructure as code that must run across tasks.

Creation begins with a weak seed that exposes primitives and required logging but has no agent loop or completion logic. The creator receives specifications, constraints and a few development cases. It must assemble a working task-family harness. The result is frozen and reused on evaluation tasks. Separating creator from executor makes runtime compatibility part of the question: a harness can behave differently when another model runs inside it.

One reported evolution trajectory makes completion concrete. Opus finds that 99 of 100 runs report success, while only 48 pass. It traces premature stopping and adds a completion check. Our original gate diagram explains the principle: inspect the requested artifact and evidence before declaring completion. It is not a reconstruction of source code or a new measured before-and-after result.

The protocol separates feedback and later evaluation. Official candidate versions must settle both the 100-task SWE-Pro and 89-task Terminal-Bench feedback legs. After development, versions are evaluated on 630 disjoint SWE-Pro instances whose scores never inform edits or final selection. The feedback pair and held-out score therefore measure different populations.

In one GPT-5.5 lineage under fixed Gemini execution, feedback rises from 56.6 to 59.1 while held-out scores fall from 42.22 to 31.90. This single trajectory shows why a visible gain is not enough. Across nine lineages, only two declared final versions are also their lineage’s best held-out version. The practical unit of progress is tested execution behavior, including the choice of which revision to keep.

Watch the companion English animation with external English, Simplified Chinese or bilingual captions.

Paper, exact arXiv v1: https://arxiv.org/html/2609.01437v1

#AIAgents #AgentEngineering #AIResearch