Harness Engineering: From Prompts to Accountable Agent Loops

A long-running agent is not “a model plus a prompt.” It is a runtime system: it selects information within a finite context, calls tools, preserves artifacts, enforces authority, manages budget and recovery, and hands results to an evaluator the agent cannot rewrite. Harness engineering is the discipline of designing that layer.

1. What This Post Does Not Do

This is not a product survey, a declaration that more subagents are always better, or a claim that RSI has arrived. The existing coding-agent architecture comparison explains how six concrete systems run. This post asks a different question: how should a harness itself be designed, constrained, and evaluated?

Two meanings must remain separate:

1. An agent harness is the runtime around a model: context, tools, state, control flow, authority, logs, and recovery.
2. An evaluation harness provisions a task, executes trials, collects evidence, and scores results. The evaluated agent must not treat it as an editable tool.

When those meanings collapse, “the agent’s score improved” becomes dangerously ambiguous. A reader cannot tell whether model, prompt, tools, environment, budget, or scorer changed.

2. Draw an Accountable Boundary First

This is not a diagram of one released system. It is a design position: the model may propose actions and the harness may schedule them, but authority and acceptance criteria cannot live only in a prompt that says “be careful.” MCP provides a protocol boundary for objects such as tools, resources, and prompts. It does not automatically make a tool response trustworthy or replace host-side policy enforcement. MCP Specification

3. Six Engineering Planes

3.1 Contract: Tools Are Not Just a List of Function Names

Tools are contracts between a stochastic agent and stateful services. An evaluable tool surface makes its input schema, authority, errors, idempotency, and response scope inspectable. The more overlapping tool paths an agent has, the easier it is to obtain a fragile result through an accidental route. Anthropic’s tool-design guidance recommends testing tools on realistic multi-step tasks and using held-out tasks to detect overfitting to development trajectories. Writing tools for agents

The testable question is not “how many tools exist?” It is whether task-shaped narrow tools reduce invalid calls, token waste, or policy violations at equal authority and budget without harming held-out success.

3.2 State: Turn Memory into Artifacts, Not an Ever-Longer Prompt

Context engineering is the selection and maintenance of token state visible at each invocation. It is not merely a larger context window, and it is not dumping every past log into a prompt. Long-running systems need task manifests, progress, tests, candidate artifacts, dependencies, and version history as retrievable external artifacts. A fresh session can then continue from the same evidence instead of guessing what a prior session did.

Three terms deserve a strict distinction:

Public long-running-harness practice emphasizes feature lists, progress files, tests, bootstrap commands, and version history so a new worker reconstructs state from artifacts rather than hidden chat memory. Effective harnesses for long-running agents

3.3 Authority: Runtime Enforcement, Not Prompt Wording

A prompt can express a preference; it cannot enforce an authority boundary by itself. A serious policy names readable and writable mounts, allowed hosts, credential scope, risky action classes, approval points, resource ceilings, network egress, and audit retention. Sandboxing and network controls are runtime mechanisms, not courtesy instructions. This is why public sandboxing designs discuss both filesystem and network boundaries. Claude Code sandboxing

One simple rule follows: an agent may propose a policy expansion, but it cannot approve that expansion itself, nor rewrite the evaluator and announce that it passed. The rule applies to browser, Computer Use, terminal, and training harnesses alike.

3.4 Control: Parallelism Is a Scheduling Hypothesis

Parallel subagents can widen research and independent execution coverage, but also multiply token, coordination, and merge cost. They fit only materially separable work with stable output contracts; shared code surfaces and tightly coupled decisions often need a common controller. A public multi-agent research system likewise reports substantial token cost for broader exploration and stresses that coordination shapes the outcome. Multi-agent research system

The parent should own global budget, decomposition, cancellation, merge, and publication. Workers should receive narrow objectives, isolated workspaces where possible, explicit deliverables, and durable result records. “Multi-agent is better” becomes an empirical claim only after comparison with a same-model, same-budget, same-task sequential baseline.

3.5 Evidence: Trace, Replay, and Rerun Are Not Synonyms

Every run should at least record harness, model, tool, environment, and policy versions; an initial-state digest; input/output and tool events; approvals and denials; artifact hashes; retries and recovery; evaluator outputs; and token, wall-clock, and resource usage. Secrets, personal data, and third-party content need redaction before retention.

That trace supports three different operations:

1. Diagnosis: why a run failed, spent unusually, or touched authority.
2. Observational replay: inspect an event record without repeating external side effects.
3. Execution rerun: execute from frozen state while declaring nondeterminism and external-dependency controls.

A video or chat transcript generally supports only the first two. It should not be called a reproducible rerun of a changing online service.

3.6 Integrity: The Evaluator Is Part of the System but Not the Optimization Surface

Deterministic tests, model graders, and human review answer different questions. Deterministic verifiers check explicit terminal states; model graders can handle nuanced text but need calibration and error reporting; humans catch unmodeled harm. Agent-evaluation engineering guidance stresses defining tasks, graders, samples, and metrics before interpreting a result as capability. Demystifying evals for AI agents

Harness changes should therefore carry a change manifest: what should improve, what must not regress, the held-out split that accepts it, and the rollback condition. Harness-Bench and VeRO are preprints exploring configuration-level or versioned agent evaluation. They provide useful methodological clues, not proof that elaborate harnesses are stronger on every task or evidence of unbounded self-improvement. Harness-Bench · VeRO

4. What a Harness Report Must Include

This table turns “the harness is good” into claims that can fail. A higher pass rate may come from hidden retries, greater concurrency, or an unequal token budget; omitting those makes the result unhelpful for engineering decisions.

5. When a Harness Tries to Improve Itself

Harness optimization can be useful: a system analyzes failed traces, proposes a small context, tool, or control-flow edit, tests it on frozen evaluation, then retains or reverts the version. It is still not automatically RSI. A stronger claim would require the modified harness to make later improvement processes reliably better on tasks that did not select it, while authority and evaluation remain outside the candidate’s control.

The minimum acceptable loop is:

1. A human or external policy freezes editable surface, budget, evaluator, and rollback rule.
2. The agent proposes a narrow edit from failure lineage and declares expected gain and regression risk.
3. An independent runner executes capability and regression splits under the same budget and records evidence.
4. An external acceptance gate merges, rejects, or reverts; the candidate harness has no write access to that gate.

This matches the boundary in the RSI note: improving one workflow is valuable, but one local gain does not establish that the system improved “the improver.”

6. A Workshop Agenda That Can Be Discussed Publicly

The following is a research agenda, not an announced event, CFP, or sponsorship notice.

1. Interfaces and environments. Tool protocols, browser and computer actions, sandboxing, portability, and deterministic verification.
2. Memory and long-horizon execution. Context lifecycles, artifact stores, checkpoints, recovery, durable jobs, and replay.
3. Governed autonomy. Capability scoping, identity, approval design, prompt-injection containment, and audit logs.
4. Optimization and evaluation. Workflow search, multi-agent scheduling, harness evolution, trace analysis, evaluator integrity, cost, and negative results.

A systems paper that claims a harness improvement should at minimum provide a versioned harness/environment manifest, task and evaluator versions, a budget ledger, trace/redaction plan, same-budget baseline, critical ablation, threat model, and failure analysis. These requirements do not guarantee correctness, but they expose a paper’s load-bearing dependencies to readers.

7. Five Falsifiable Research Questions

These questions join AutoResearch, Agentic RL, and Agent Research Environments: all require separate boundaries for what can change, what feedback is observed, and what finally counts as evidence.