AutoResearch: From a Falsifiable Research Loop to the Evidence Map

The interesting part of AutoResearch is not a claim that an agent has suddenly become a scientist. It is the discipline of turning research into an auditable loop: propose a change, run an experiment, read evaluator-owned evidence, retain or reject an artifact, and choose the next experiment. That loop is useful. By itself, however, it is neither a benchmark nor evidence of recursive self-improvement (RSI). This long note makes one continuous argument in five parts: Part I is the methodological backbone of AutoResearch; Part II is a Deep Research Agent's discovery–qualification–evidence–citation protocol; Part III converges RSI, Auto Research, long horizon, RL environments, and synthetic-instance training into one evidence map; Part IV is a research roadmap that turns project progress into auditable evidence; Part V constrains nine adjacent hot directions back to the five core questions.

Part I · AutoResearch: Turning a Research Loop into a Falsifiable System

I-1. Four Things That Must Stay Separate

Karpathy’s autoresearch is a clean prototype. The agent edits a constrained training script while preparation and evaluation stay frozen; short runs return a metric and the loop keeps better candidates. It demonstrates bounded empirical optimization. It does not claim cross-hardware comparability, general scientific competence, or RSI.

I-2. What Sits Between Code Editing and Research

A research loop needs answers to four questions:

1. What is the intervention? Which files, hyperparameters, tool calls, or experimental designs may change?
2. Where does evidence come from? Visible development metrics, a sealed verifier, physical constraints, or human feedback?
3. What is retained? Only the final best artifact, or hypotheses, failed trials, and a decision lineage?
4. How is local optimization ruled out? Is the evaluator frozen, and are selected results rerun on held-out seeds, settings, or an independent runtime?

Without the fourth answer, the honest label is usually “optimization of a local harness,” not research. A stronger claim puts evaluator code, data preparation, and hidden tests behind a write boundary, then reruns winners on preregistered held-out settings.

I-3. Scientific Tasks Can Measure Different Objects

This is not a ranking of which system is “more scientific.” It first says that paths to a high score cannot borrow one another’s conclusion: reproducing a paper is not proposing a new mechanism; implementing an expert-written extension is not choosing a valuable hypothesis; and searching better in one environment is not surviving a new environment or laboratory. ResearchClawBench builds tasks from real papers, related literature, and raw data, but keeps the target paper on the evaluation side and scores research artifacts with expert rubrics. That makes re-discovery of a known target more inspectable than freely writing a report, but it is still neither an independent reproduction of the original claim nor an open-ended discovery. PaperBench notably executes the submitted codebase in a fresh GPU container, while CORE-Bench focuses on computational reproduction from supplied code and data. They make “it runs” more checkable, but they do not make the original paper automatically true or a reproduction task a scientific discovery. FML-Bench separates strategy from execution infrastructure and records process metrics, which warns against attributing a stronger harness, more parallelism, or cheaper API calls to a “better research strategy.” Terminal-Bench Science is still under construction and should not be cited as a released, complete benchmark.

I-4. Candidate Search Cannot Also Accept the Conclusion

The development phase of AutoResearch is inherently adaptive: the next hypothesis follows a previous metric, log, or failure. That is not a flaw, but it means the best candidate selected from many trials cannot be certified by the same feedback channel. Exploration and acceptance need two loops with different boundaries.

Generalization in Adaptive Data Analysis and Holdout Reuse shows that repeatedly viewing the same holdout adaptively can overfit the holdout itself; The Generic Holdout further considers separating unrestricted exploration from limited-exposure acceptance. These are conditional statistical mechanisms, not an automatic guarantee supplied by the engineering habit of hiding one test set. The acceptance protocol proposed here is therefore an auditable engineering minimum only: record development/acceptance query counts, freeze a candidate before submission, limit final feedback, and rerun it in a clean runtime. A formal generalization guarantee requires a method or fresh holdout appropriate to the task and query regime.

This gives a more honest result sentence: “A candidate selected after q development queries on D_dev was frozen as hash h; it met preregistered postconditions on selection-independent H_accept and an independent run R.” That still does not establish novelty, causal explanation, or publication, but it creates objects another reader can inspect and refute rather than merely trust as the “best run.”

I-5. Frontier Auto Research: The Deliverable Is a Reviewable Evidence Package

Frontier Auto Research (hereafter ART) is a working name here for a proposed research program, not a released team, paper, or leaderboard. It should not compete with EdgeBench, AutoLab, or MLS-Bench for the label “more research-like.” It changes a more concrete unit: a task’s deliverable is not only a score or final code, but a narrow claim and evidence package that an external reader can try to refute.

The question “can solving one task lead to a top-tier conference paper?” should therefore become a more honest criterion: can the task produce a minimal but complete research evidence package? It needs a clear problem, preregistered edit surface, executable artifacts, reasonable controls, negative results or failure analysis, held-out replication, and a scope-limited explanation. Publication is an external peer-review decision, not an environment reward.

This definition also avoids a false conflict with EdgeBench. EdgeBench asks how sustained interaction in real environments becomes a better artifact and learning curve. ResearchClawBench asks whether a preconstructed hidden-target task can re-discover parts of a research artifact. The proposed Frontier Auto Research question is whether one improvement supports a conclusion an independent reader would examine. They can use related harnesses while answering different questions. MLS-Bench, AutoLab, EdgeBench, and ResearchClawBench respectively provide reference points for method generalization, long-horizon closed-loop optimization, within-environment learning, and hidden-target re-discovery, not baselines that need replacing. MLS-Bench · AutoLab · EdgeBench · ResearchClawBench

I-6. A Research Card Worth Publishing

I would report a serious AutoResearch experiment with this card, not a single attractive trajectory:

The pivotal ablation removes experimental lineage while keeping model, tools, and budget fixed. Only when the advantage disappears is there evidence that the loop learned from its history rather than merely sampled a lucky candidate.

I-7. A Research Evidence Package Is Not One Log: Five Non-Interchangeable Kinds of Evidence

Generated code, a completed notebook, or a written report do not by themselves make an agent result reviewable research. Recent work sharpens the decomposition. ResearchClawBench uses hidden target papers and expert rubrics to make re-discovery of target work a scoreable research-artifact task, but its task package and rubric are a measurement protocol, not a decision about the target paper’s truth or a new discovery. SciAgentArena uses interactive, stepwise-verifiable tasks to measure well-specified scientific workflows, while reporting that current agents remain uneven on open questions and novel insight. AutoResearchBench separates literature discovery into deep target finding and broad collection. ARA extracts a workflow graph of sources, methods, experiments, and outputs from papers. SocSci-Repro-Bench explicitly separates agent failure from source materials that are themselves not reproducible.

EurekAgent treats permissions, artifacts, budget, and human oversight as separate environment-engineering planes for research agents. Meta-Agent Challenge gives a meta-agent a sandbox, time limit, and held-out evaluation API to develop an agent artifact, with reward-hacking defenses in the framework. They support treating a research harness as an evaluable object; they do not permit a sandbox or generated agent artifact to be called verified strong RSI.

I-8. ART’s Minimum Acceptance Protocol: From Task Completion to a Reviewable Claim

Frontier Auto Research should split task completion into the following five gates. They are proposed design, not reported results from a released system.

These gates also give safer wording. Completing a computational workflow is workflow execution. Finding qualifying papers is literature discovery. Rerunning an existing result is reproduction. Only when controls, held-out verification, and a scope explanation all hold may a report say evidence-supported empirical claim. A paper, simulated review, or one scalar score is not a substitute for those gates.

I-9. Cost Is Not a Model Price List

When a research loop uses expensive closed models, multiple workers, or a GPU cluster, cost is an experimental condition. Do not report only the selected best run, and do not treat a public price on one day as a fixed property of a model. A reliable ledger separates immutable measured usage (tokens, requests, GPU/environment time, retries, concurrency, and human minutes), period-level invoiced cost, and a dated, currency-specific price-snapshot estimate with its allocation rule and source. The official Usage API describes project-level usage primitives, but experimental reporting still has to include environments, failures, and concurrency. OpenAI Usage API

For the question whether an Opus- or GPT-class model is too expensive, the right answer is not a guessed total first. Freeze a budget and report attempts, completions and cancellations, cached and uncached tokens, model requests, browser or sandbox minutes, GPU active/reserved time, queue time, external tools, human confirmations, and total development cost for the same research conclusion. Readers can then distinguish an algorithmic gain from more parallelism, retries, or money.

I-10. When RSI Is an Appropriate Claim

RSI concerns the system that performs the next round of research, not merely the current target. A system that changes its own tool selection, context policy, search program, or agent policy, then performs better on sealed tasks it did not select against, approaches empirical evidence for RSI. Darwin Gödel Machine and AlphaEvolve are serious directions to study; their experimental results are still not a proof of unbounded recursive improvement.

I therefore use deliberately conservative labels:

• Improving an external artifact: automated target optimization.
• Reusing experimental history to improve continuously: AutoResearch / autonomous research loop.
• Improving the agent or harness that runs future loops, with gains preserved on sealed, shifted evaluations: empirical evidence for RSI.

In addition, Auto Research with Specialist Agents is a reminder to report gains from “specialist-agent division of labor” separately from simply spending more compute, which is the same discipline as the lineage ablation above.

Part II · Deep Research Agents: From Retrieval to Reviewable Claims

Part I separated exploration from acceptance; the loop’s input side—literature and web evidence—needs the same discipline. Deep Research is not “search more pages, then write a longer summary.” A reliable research agent must separately find candidate sources, verify that a source applies, extract evidence from a specific version, bind each claim to an appropriate citation, and preserve a contamination boundary between web retrieval and evaluation. The real deliverable is not a bibliography, but a set of narrow claims that can be traced, challenged, and updated. The central claim is: retrieval != qualification != evidence extraction != citation != accepted claim—doing any one well does not replace the next.

II-1. Separate Five Objects First

AutoResearchBench divides scientific literature discovery into two tasks: Deep Research, which traces a specific target paper through progressive probing, and Wide Research, which collects a set of papers meeting stated conditions. SAGE makes scientific retrieval a reasoning-intensive benchmark and finds that evaluated deep-research systems still struggle. Writing multiple queries therefore does not show that the retrieval process found evidence meeting a research condition; query formulation, corpus, retriever, and reranking policy are all part of the result.

II-2. A Reference List Is Not Ground Truth by Default

Literature search is often scored against a human reference list, but that does not make the list complete, unbiased, or strictly relevant to the current question. Rethinking Literature Search Evaluation argues for reporting recall, topical relevance, ranked-list diversity, and diagnostics such as co-authorship distance separately rather than treating one human list as the only truth. This does not license an agent to dismiss expert citations. It requires the system to state what it optimizes, what it misses, and which protocol decides relevance. A personal knowledge base should therefore not write “cited by a paper” as “verified.” A stronger entry separates:

1. Bibliographic facts: title, authors, version, publication date, persistent URL;
2. Scope facts: object of study, tasks, data, models, environment, and evaluator;
3. Empirical claims: what was observed under which conditions;
4. Interpretation or inference: the design implications we draw, explicitly labeled as inference;
5. Conflict and unknowns: opposing evidence, missing experiments, preprint status, and items to recheck.

II-3. Build the Personal Knowledge Base from Source Cards and Claim Ledgers

A lightweight but rigorous local knowledge base does not require a complicated database. It needs traceable objects, rather than isolated summaries:

SourceCard(
  source_id, canonical_url, title, authors, source_type, status,
  version_or_date, accessed_at, scope, methods_or_interface,
  findings, limitations, quoted_or_locatable_evidence, content_digest)

ClaimRecord(
  claim_id, text, claim_type, scope, source_ids, support_type,
  uncertainty, conflicts, author_inference, last_checked_at)

ResearchDecision(
  question, alternatives, evidence_claim_ids, rejected_evidence,
  decision, rationale, owner, review_or_recheck_trigger)

support_type should at least distinguish direct finding, source metadata, implementation/documentation fact, and our inference. The last kind is valid, but must not masquerade as a paper result. For dynamic projects, version_or_date and last_checked_at are not decoration: task counts, leaderboards, APIs, and repository interfaces change.

II-4. On the Web, Content Is an Evidence Candidate, Not an Instruction

Web research creates two distinct risks. First, webpages, PDFs, search snippets, and tool output may contain untrusted or malicious instructions. They can enter the evidence-candidate channel, but cannot alter the agent’s authority, hidden rules, or evaluator. Second, a public benchmark’s questions, metadata, or answers can be retrieved at test time. Search-Time Contamination in Deep Research Agents separates benchmark-metadata, question-context, and explicit-answer leakage, and recommends isolated search, transparent trajectories, and controlled benchmark access.

Web retrieval also does not make one-agent systems automatically preferable. AgentWebBench treats coordination between a user agent and website content agents as its own problem and separately observes retrieval, synthesis, and interaction reliability. The direct lesson for multi-agent research is that workers can widen query coverage, but each worker’s sources, authority, deduplication, conflicts, and merge rule still belong in the evidence ledger.

II-5. A Deep Research Protocol That Can Fail

The table below is a minimum protocol for personal research, ART’s literature phase, or a future benchmark. It is a design proposed here, not a result claim about an existing system.

Metrics should remain separate as well: discovery recall/coverage, qualification precision, claim-level citation entailment, citation diversity, duplication/conflict rate, contamination-hit rate, and time/cost per acceptable claim. No single score establishes that a system “researches well.”

II-6. Toward the Five Core Threads

The smallest practical conclusion is simple: turn every important sentence into a ClaimRecord with a source, scope, status, and recheck time before deciding it belongs in a post, research design, or training data. Then the knowledge base is not a warehouse of links. It is an evidence system that can support the next retrieval, refutation, and experimental decision.

Part III · Research Agenda: Five Threads and the Evidence Map

Parts I and II fixed the discipline of the experiment loop and of retrieval; this part converges the program into five distinct, rejectable artifacts: controlled self-improvement (RSI), reproducible research artifacts (Auto Research), recoverable long-horizon execution, auditable environments and evaluators, and synthetic training experience that transfers to unseen work. A credible research-agent program must state what each layer owns; a positive result in one layer cannot vouch for another. The tables below bind each primary source to the narrowest conclusion and boundary it directly supports.

III-1. One Program, Five Non-Interchangeable Layers

This also answers the division of work. Frontier Auto Research / ART belongs to the research-artifact layer. ClawBench V2 and WebsiteBench belong to the environment layer. RewardHarness -> RSIBench -> OpenRSI belongs to the candidate-change and acceptance layer. Synthetic instances and agentic RL form a cross-cutting training layer. A long-horizon harness runs through every system that needs multi-stage work. The threads are dependencies, not a ranking: the environment fixes observation, action, and a credible reward; a long-horizon harness determines whether an agent can actually use that feedback; synthetic data and RL determine which experience reaches a model; Auto Research determines whether the improved external object can support a research conclusion; RSI then asks whether the improver itself became better under strict conditions.

III-2. Enter from the Matching Guide: Five Threads and Adjacent Directions

III-3. RSI: A Candidate May Modify; the Acceptance Gate May Not Be Candidate-Controlled

Convergence for RSIBench / OpenRSI. A candidate may write only to a declared editable surface; development metrics, the hidden acceptance evaluator, permission policy, logs, and the release decision live in another write-authority domain. A multi-generation curve fixes a resource policy and reports rejected changes and rollbacks. Report candidate_gain, acceptance_gain, heldout_gain, regression_rate, and integrity_violations separately, plus four independent curves: improvement operator, selection/acceptance, transfer/persistence, and integrity/reversibility. Only the final two together justify discussing an “improver that improves.” Without them, “self-optimization experiment” or “harness search” is more accurate than RSI.

III-4. Auto Research: Improving a Research Artifact Is Not Automatic Science

Convergence for ART. Frontier Auto Research should not compete with AutoLab by “optimizing longer,” with MLS-Bench by “having more method tasks,” or with EdgeBench by “having a longer environment.” Its unit of contribution is a minimum research package: problem and assumptions; runnable baseline; declared editable surface; experiment graph; strong controls; negative results; rerun script; data/code versions; cost ledger; and one narrow conclusion supported by held-out or fresh settings. It may produce a paper candidate; novelty and field value remain decisions of peer review.

III-5. Long Horizon: State, Recovery, Planning, and Safe Degradation

Long-horizon harness acceptance. Preserve a task manifest, initial-state hash, progress ledger, checkpoints, tool/model/environment versions, recovery causes, retries, human takeovers, and final evaluator record. The core metrics are not one pass rate but outcome, recovery_success, state_reconstruction, unsafe_action_rate, cost_per_success, and time_to_stable_artifact. Terminal-Bench’s home, TB2 benchmark page, TB3 status, TB Science status, plus Terminal-Bench Challenges and its run guidance should be read by their released status: beyond final-state verification, the process remains diagnostic and reproducibility evidence, not disposable noise.

III-6. Environment: Before Training, Give Reward an Auditable Origin

Convergence for ClawBench V2 / WebsiteBench. Write five planes as versioned contracts: reset (backend, identity, browser/VM state); observe (DOM/CDP, a11y, screenshot, or terminal); act (authority, egress, intercepted commit surface); verify (independent judge and hidden rules); and evidence (append-only traces, artifact hashes, network/request records). For every reward, include at least four counterexamples: right endpoint/wrong payload; injected page text; plausible screenshot/no server submission; and missing or reordered trace evidence. SForge’s work/judge separation (SForge repo, Harbor docs, ClawBench repo) is an isolation pattern worth studying, not proof that isolation has been solved. Interception controls only its declared browser/CDP/proxy plane and cannot claim every OS or network side channel.

III-7. Synthetic Instances and Foundation-Model Training: Scale Is Not Evidence, Extrapolation Is

Convergence for training. “2K synthetic instances” is only a falsifiable scale hypothesis. Before training, record provenance: source, license, PII review, generator/checkpoint, templates, and environment image. During training, compare SFT-only, RL-only, SFT+RL, and frozen-base baselines. After training, accept only on tasks split away by website, workflow, account template, semantics, and evaluator rules. Report accepted/rejected trajectories, deduplication, difficulty/diversity, reward distribution, tokens, GPUs, sandboxes, APIs, wall clock, and human review. Without that, “synthetic instances improved a foundation model” has no auditable object.

III-8. Choose the Question Before the Evidence

This map does not rank “who is best.” It routes by question first, then binds each link to the one cell it directly supports.

III-9. Turn Project Differences into Testable Designs

Per-project core objects and decisive tests are given in the project map of IV-3. That table deliberately has no “best model” column. Compare task distribution, execution interface, hidden information, reward, resource budget, training status, and evaluation split before comparing scores; without those conditions, environment differences become falsely reported as model differences.

III-10. A Reusable Citation and Reporting Protocol

Every research note or blog conclusion should carry these labels:

1. Source type: classic theory, peer-reviewed / preprint, official documentation, implementation, or future-work proposal.
2. Object: external artifact, policy, harness, environment, verifier, evaluator, or the mechanism that improves them.
3. Evidence form: theorem, controlled experiment, runtime interface, task specification, or design recommendation.
4. Boundary: model, task, harness, budget, data/split, version, and access authority.
5. Falsifier: what observation weakens the claim, such as changing the verifier or task distribution, a gain vanishing at equal budget, or the evaluator being contaminated.

For literature and web research, also retain a SourceCard and a ClaimRecord (schema in Part II). For entry points into the companion posts, see the table in III-2.

Part IV · Research Roadmap: Making Agent Progress Auditable

The evidence map settles what to believe; the remaining question is what to build. A worthwhile agent-research program is not a collection of benchmark names or one attractive demo. It must answer a harder question: what did a system actually change, why should we believe the change helped, and how do we know it did not merely learn to influence the evaluator? This part places Frontier Auto Research, ClawBench, RewardHarness, RSIBench / OpenRSI, and WebsiteBench on one map. The unifying aim is research-agent systems that preserve evidence, admit refutation, and learn safely.

IV-1. Separate Nine Kinds of Claims First

Research blogs become unreliable when preprints, code, internal plans, and intuitions are written in the same voice. I use nine labels rather than collapsing them into “progress.” This taxonomy adapts COPE’s guidance on preprint transparency, ACM’s distinction between available artifacts and validated results, and ICMJE’s principle of disclosing direct support and a sponsor’s role.

Pre-publication check: attach an accessible primary-source link to every statement that already happened, recording version, date, and scope; every result sentence should point to a table, run, or public artifact, otherwise write an observation or pending validation; mark internal names as working labels at first use and invent no team, leaderboard, user count, compatibility guarantee, or submission status; an algorithm comparison must state the fixed model, harness, data split, budget, seeds, and metrics; describe a workshop only as a theme or proposal unless a public CFP exists; name a collaborator or sponsor only when confirmed, disclosing the funder’s role and restrictions; check permissions, de-identification, and provenance for every external figure, screenshot, trajectory, and live-web datum; and fact-read the Chinese and English versions separately so numbers, status words, qualifications, and negations are semantically equivalent.

IV-2. One Spine: Action, Learning, and Evidence

I divide a research agent into three mutually constraining layers: action environment (an agent produces a deliverable artifact in a terminal, browser, or computer interface; irreversible actions need explicit safety boundaries); learning and harness (the model, context, tools, memory, and control loop jointly determine actual behavior, and training should not reduce the deployed harness to a different one-turn prompt); and evidence and evaluator (evaluation, logs, held-out data, reruns, and a cost ledger must sit outside the agent’s write boundary). The unit of work is therefore not “an agent completed a task,” but an evidence package: problem, permitted interventions, artifact, evaluator version, complete lineage, held-out evaluation, and resource ledger. Without these seven pieces, a result can guide debugging, but not sustain a research conclusion.

IV-3. Project Map: Public Foundations, Working Goals, Decisive Tests

IV-4. Frontier Auto Research Is Not a Longer Run

“Automated research” is easily mistaken for “letting an agent run for a long time.” Duration is not enough, nor is it the key distinction. autoresearch demonstrates a constrained editable surface, fixed experiment duration, and keep-or-revert loop. AutoLab measures improvement of a correct but suboptimal baseline within a time budget. MLS-Bench measures whether an ML-method mechanism transfers across settings. EdgeBench studies how long-horizon environmental feedback changes a best-so-far artifact. All matter, but they measure different objects. The proposed distinction for Frontier Auto Research is the scholarly product of a task.

The honest requirement is narrower: if a task is called a research task, it should at least permit a reviewable contribution form, including a clear question, reproducible experiment, reasonable controls, scoped conclusion, and inspectable limitations. Publication remains a peer-review judgment, not an environment reward.

IV-5. Web Agentic RL: Protect the Outcome Before Choosing an Algorithm

Web RL has a sharp difficulty: the agent can observe pages, generate text, and call tools, while potentially affecting logs, caches, form payloads, judge prompts, or the training distribution. A high reward is not enough evidence that a user’s intent was completed. ClawBench’s current public two-stage protocol is a useful direct example: first establish that the final request was intercepted, then ask a judge whether its body satisfies the instruction; only both conditions yield success.

The more general form of reward tampering is an agent’s instrumental incentive to influence the reward function or its inputs. The causal analysis by Everitt et al. is not a web paper, but it yields a portable design question: can the policy raise reward by changing the evidence being assessed, without changing the true outcome? If it can, that evidence channel must leave the agent’s control domain. A related failure picture is catalogued in Google DeepMind on specification gaming.

IV-6. PPO, GRPO, and 2K Synthetic Tasks Are an Experimental Design, Not a Conclusion

The public Polar preprint proposes proxying LLM API traffic, recording token-level interactions, and reconstructing trajectories aligned with the real harness; its reported agent-training experiments use GRPO on software-engineering harnesses. That does not mean any browser harness is already trainable by RL, nor that PPO or GRPO has won on the web. The smallest credible subsequent web-RL experiment should start with a falsifiable matrix, not a large narrative:

PPO does not win merely because trajectories are long: it needs a credible critic and useful intermediate information. GRPO does not win merely because it avoids a critic: if every rollout in a task group succeeds or fails, its relative signal is near zero. The correct sentence is not “PPO may be better than GRPO”; it is: measure reward resolution and credit assignment first, then let the algorithm comparison answer the question.

IV-7. Cost Is Also a Result

The cost of a long-horizon agent is not an API sticker price; the ledger structure is as defined in I-9. Model and training choices should report a Pareto curve rather than only the top score. This post intentionally does not quote specific provider or model prices: availability, limits, and pricing change, and should be fixed independently in a timestamped cost ledger for each experiment.

IV-8. Two Workshop Themes Worth Taking Seriously

These are research themes, not event or sponsorship announcements. First, Auto Research: which benchmarks, methods, and environments can support hour- and day-scale research loops? How can frozen evaluators, held-out splits, evidence lineage, and anti-reward-hacking become a default protocol? Into which falsifiable capability claims should “AI Research Scientist” be decomposed? Second, Agent Harness Engineering: agent capability is not only in a prompt; it also lives in context, file state, tool permissions, loops, concurrency, recovery, evaluation, and training interfaces. Lilian Weng’s Harness Engineering for Self-Improvement places workflows, persistent state, subagents, evaluation, permissions, and self-improvement in one systems view; this post follows the same discipline while centering how each loop leaves evidence an external reader can audit.

IV-9. What to Prove Next, Rather Than Announce

1. Write a pre-registered task card for each line: objective, artifact, editable surface, risky actions, visible feedback, hidden feedback, budget, held-out split, and stop condition.
2. Keep append-only experimental lineage: every hypothesis, diff, runtime, result, selection, and revert receives a checkable fingerprint.
3. Isolate evaluator from training policy: task rules, reference traces, judge prompts, and final tests live in separate access domains and every version change receives regression tests.
4. Report four tables: true outcome, safety, cost, and transfer. A single reward is a diagnostic, not the end of the story.
5. Keep RSI conditional: only a system that improves the mechanism of future improvement, with gains retained under frozen and distribution-shifted evaluation, earns a discussion of empirical RSI evidence.

Part V · Hot Adjacent Directions: Nine Paths Back to the Core Five

V-1. Overview

The roadmap converges on five threads, yet agent research still expands too easily around fashionable terms: memory, multi-agent systems, MCP, world models, browser and computer use, and self-evolving skills. Each matters, but none should automatically become an unbounded new project. This part compresses 36 paper/preprint sources plus two official runtime documents into nine adjacent directions and asks one question: how can they make one of the five core research threads more measurable, safer, or easier to falsify? (The count 36 covers distinct arXiv paper/preprint citations only; OpenAI Computer Use and Browser Use Terminal are two additional runtime documents; the skills section’s SkillRouter and Skill Retrieval Augmentation are supplementary routing evidence.)

V-2. State and Memory

Agent Memory separates construction, retrieval, and generation phases in stateful, tool-using long-horizon workloads; MemGym isolates memory measurement across deep research, coding, and computer use; AgeMem treats long- and short-term memory operations as trainable actions; MAGE organizes execution state as a hierarchy that can grow, compress, maintain, and revise. Write-back: a long-horizon harness should version task manifest, external facts, candidate artifacts, unresolved hypotheses, checkpoints, and reasons for state changes, reporting cost for each retrieval, compression, and recovery; RSIBench can treat a candidate change that breaks state reconstruction as a regression. Boundary: retrieving a correct snippet on one task does not show a system maintains causal long-term state.

V-3. Multi-Agent Systems

SABER examines operational safety in stateful coding workspaces; NRT-Bench uses replayable multi-turn red-teaming for team risk; MAFBench finds orchestration choices can change accuracy, latency, and coordination; MAESTRO exports framework-agnostic traces and system signals. Write-back: ART and ClawBench V2 baselines should include one agent, sequential multi-step work, fixed parallelism, and selector-driven delegation under fixed total tokens, wall-clock, worker-hours, and authority; each subtask’s read/write grant, input summary, artifact, and merge decision belongs in the trace. Boundary: parallelism may only move expenditure to more API calls or wider authority.

V-4. MCP and Tool Governance

Governance Gaps discusses missing governance primitives; From Tool Connection to Execution Control argues runtime execution control should not be replaced by a connection protocol; Security Threat Modeling provides a protocol threat-modeling lens; MCP-Persona shows tool portfolios can be evaluated against user or task preferences. Write-back: beyond a tool name, an environment contract should declare schema, identity, authority, idempotency, external effects, audit events, revocation, and error semantics; an RSIBench candidate must not modify the verifier, policy, or credential routing. Boundary: MCP or A2A are interoperability interfaces, not sandboxes, least privilege, approval workflows, or complete audit systems.

V-5. Agent Security

BrowseSafe studies prompt injection in web content; Security Considerations for AI Agents separates input, model, and sandbox attack surfaces; AgentDyn offers a dynamic real-world injection-testing direction; WASP studies web-agent injection attacks in isolated environments. Write-back: ClawBench V2 / WebsiteBench should label untrusted pages, tool responses, files, and task text separately, with the four counterexample classes of III-6 plus deceptive tool descriptions. Boundary: a guard’s detection rate is not safety; report false blocks, misses, overhead, attack-distribution shift, and authority escalation.

V-6. Browser / Computer Use

BrowserGym puts multiple browser-task settings behind a common Gym-facing interface; OSWorld binds open-ended desktop tasks to configured computer environments and executable evaluation; the OpenAI Computer Use guide makes the screenshot-action loop, isolated browser/VM, human confirmation for high-impact actions, and untrusted page input harness responsibilities; Browser Use Terminal exposes profiles, secrets, domain policy, resumable sessions, and event logs as parts of a browser runtime. Write-back: a task manifest should version at least observation_surface, action_surface, account and credential scope, reset/replay tier, domain/egress policy, action receipts, sidecar evidence, and a non-agent-writable backend postcondition. Boundary: a screenshot or click does not represent a backend commit, and real-account side effects are not necessarily replayable. Fuller substrate comparison in Web-Agent Environments and Agent Research Environments.

V-7. World Models and Embodied Agents

WorldArena highlights a gap between visual quality and interactive utility; WorldArena 2.0 extends evaluation toward visuotactile and interactive RL; WorldLines binds long-horizon embodied tasks to traceable state, memory, and planning; LongAct studies dependencies, memory, and adaptive planning in free-form household tasks. Write-back: synthetic instances should generate not only task text but initial state, executable transitions, failure branches, and postcondition checks. Boundary: realistic rendering does not establish environment semantics, causal transitions, or out-of-distribution control.

V-8. Harness and Planning

Agentic Harness Engineering organizes editable components, distilled experience, and decision predictions into a falsifiable harness-evolution loop; Agent Planning Benchmark separately measures planning, feedback-conditioned replanning, noisy tools, and unsolvable tasks; TRACE emphasizes trajectory utility, evidence grounding, and the minimum support needed for success; ClawArena-Team scores subagent management together with least privilege and modality routing. Write-back: every public result should be attributed to model + harness + environment + tool policy + evaluator + budget. Boundary: a longer prompt, more complex loop, or higher outcome cannot automatically be attributed to model capability or genuine planning.

V-9. Large Tool Spaces and Orchestration

PlanBench-XL tests retrieval, adaptation, and recovery in a large tool ecosystem with tool failures; SAGA treats the full agent workflow as the scheduling unit; Uno-Orchestra studies selective delegation; AutomationBench combines cross-application API discovery, policy adherence, and final writes. Write-back: a multi-tool environment should expose discovery, invocation, verification, recovery, and stopping, reporting cost_per_verified_success, tail latency, failure recovery, invalid calls, and authority escalation. Boundary: lower latency or greater throughput is not correct completion.

V-10. Continual Learning and Skill Evolution

ACuRL connects target-environment exploration, curriculum task synthesis, and continual adaptation; MUSE-Autoskill treats creation, storage, management, testing, and revision as a skill lifecycle; Agent Skill Evaluation and Evolution surveys evolution through execution feedback, trajectory distillation, compression, and RL; SkillOpt constrains skill-document edits to changes accepted by held-out score; SkillRouter and SRA-Bench separate candidate retrieval, whether loading is needed, and subsequent use as distinct routing problems. Write-back: a skill needs code-like versioning (inputs/outputs, authority, tests, applicable distribution, cost, expiry); RSIBench can use accepted and rejected external-skill edits as a controlled self-modification surface. Boundary: a self-edit loop sharing its generator, task template, or reward may merely fit locally.

V-11. Converge to Five Design Decisions

Hot directions should become constraints and diagnostics for the five core questions, not nine simultaneous product tracks. The next iteration should implement the smallest tasks and counterexamples capable of testing these decisions, then decide which line merits expansion into a paper or training program.

Where to Go Next

1. Environments: what artifacts, feedback, and verifiers different research tasks actually hold an agent accountable for.
2. Agent Planning (in Long-Horizon Agents): make research goals, hypotheses, evidence gaps, controls, budget, replanning, and stopping conditions into an inspectable plan lineage.
3. Agentic RL (in Agent Harness Engineering): why a training system must preserve the behavior of the real harness.
4. Long-horizon agents: put checkpoint, recovery, failure, and cost from research runs on one evidence trail.
5. RSI: when it is justified to move from optimizing a target to improving the improver.