Agent Research: A Reading Map from Questions to Evidence

This is not a leaderboard of “who is best.” It is a navigation map for research. Agent systems often have similar names while answering different questions: some improve an external research artifact, some learn through an environment over time, some connect a real harness to RL, and some claim to improve the mechanism of improvement across generations. Separate the question first, so citations, experiments, and conclusions do not borrow strength from one another.

1. Choose the Question Before the Evidence

2. Four Reading Paths

A. Auto Research: Research Artifacts, Not an “AI Scientist” Label

Start with MLS-Bench to see why a “new method” needs evidence across settings and scales. Then read AutoLab for persistent artifact optimization under a fixed wall-clock budget. Finish with EdgeBench to separate a final score from a learning curve during a run. Their shared feature is executable feedback; their claim objects differ: method generalization, an artifact-optimization loop, and environment-learning speed.

Frontier Auto Research, later extensible into ART, should therefore not aim to duplicate a benchmark score. Its target is a concrete research question and an empirical claim strong enough for a top-tier paper: independent reproduction, a cost ledger, and fresh validation. At present, that is a research scope and an acceptance standard, not a released benchmark or an “automatic publication” capability.

B. RL Environments: Freeze Something Before Claiming Learning

Terminal-Bench 2.0 puts final-state verification into realistic terminal workflows. TB3 and Terminal-Bench Science are works in development and should be read according to their released status and contribution rules, not as completed task counts. Harbor supplies the interface vocabulary of task, environment, agent, verifier / reward, and rollout. SForge’s work / judge separation is an isolation pattern worth studying, not proof that isolation has been solved.

For web environments such as ClawBench V2 or WebsiteBench, the minimum reportable unit is not “a browser sandbox.” It specifies how backend state resets; which actions are intercepted before commitment; network and account authority; which artifacts the agent writes; which evidence an independent sidecar collects; and whether the verifier lives in a domain the candidate policy cannot write. This design should report ClawBench’s recording and control ideas separately from the browser and computer interfaces.

C. Agentic RL: The Training Interface Is Part of the System

Read DeepSeekMath first for GRPO at the objective layer, then Agent Lightning for decoupling execution from training and its trajectory-attribution interface. Next, read Polar for recording token-faithful trajectories in native harnesses with compatible model APIs. Finally, read AgentTrek, placing GUI trajectory synthesis in the category of offline data rather than mistaking it for online RL.

For RL research on ClawBench V2, this means PPO, GRPO, or any other algorithm is only one layer. A positive result must also fix the rollout harness, visible state, action authority, reward-to-loss mapping, independent evaluator, train / validation / test split, and token, GPU, API, and wall-clock costs. Neither a harbor-native task nor a synthetic-trajectory corpus alone constitutes a positive RL result.

D. RSI: A Candidate May Modify; the Acceptance Gate Cannot Be Candidate-Controlled

Start with I. J. Good’s conditional argument and the formal model of Gödel Machine, then read bounded empirical systems such as Darwin Gödel Machine and the Red Queen Gödel Machine. Return next to Reusable Holdout and reward-tampering evidence: repeated selection on one evaluator can overfit, and once a candidate can affect a reward channel, the score itself is an attack surface.

The reasonable direction for RewardHarness → RSIBench → OpenRSI is to turn this boundary into a task contract. A candidate proposes changes only within a declared editable surface; development, acceptance, and final held-out evaluation are separate; graders, hidden tasks, resource ledgers, and release decisions are not candidate-writable. This is a falsifiable protocol draft, not a declaration that strong RSI has already been achieved.

3. Turn Project Differences into Testable Designs

This table deliberately has no “best model” column. Compare task distribution, execution interface, hidden information, reward, resource budget, training status, and evaluation split before comparing scores; without those conditions, environment differences become falsely reported as model differences.

4. A Reusable Citation and Reporting Protocol

Every research note or blog conclusion should carry these labels:

1. Source type: classic theory, peer-reviewed paper / preprint, official project documentation, implementation, or future-work proposal.
2. Object: external artifact, policy, harness, environment, verifier, evaluator, or the mechanism that improves them.
3. Evidence form: theorem, controlled experiment, runtime interface, task specification, or design recommendation.
4. Boundary: model, task, harness, budget, data / split, version, and access authority.
5. Falsifier: what observation weakens the claim, such as changing the verifier or task distribution, a gain vanishing at equal budget, or evidence that the evaluator can be contaminated.

This makes the blog useful both for learning and for research. A reader can return from a claim to its source; a researcher can trace from a conclusion to the experiment it requires. See the companion detail posts on AutoResearch, Agent Research Environments, Agentic RL, and RSI.