Language / 语言

研究型 Agent 纲领：五条主线，32 个新增证据位

Jul 2026 · RSI / Auto Research / Long Horizon / Environments / Synthetic Training

第二轮调研的结论不是再添加一串 benchmark 名字，而是把研究收束成五种不同的、可被拒绝的产物：受控自我改进、可复现的研究工件、可恢复的长程执行、可审计的环境与 evaluator、以及能迁移到未见任务的合成训练经验。一个可信的 research agent program 必须同时说明这五层各自负责什么，不能让一层的正向结果替另一层背书。

证据状态：以下 32 个新增槽位来自论文、公开 benchmark 或官方项目材料，许多仍是预印本。表中只写其最窄的可用结论。ART、ClawBench V2、RSIBench / OpenRSI 与 WebsiteBench 在本文仍是研究方向或工作名，除非另有明确公开发布；本文不报告它们已有的训练分数、论文接收或产品状态。

1. 一个 program，五个不能互换的层

层 要回答的问题 该层真正的输出 对下一层的约束
RSI 改进过程本身是否跨代变好？ 版本化候选、独立 acceptance、回退和跨代曲线 候选不可写 acceptance gate 或其证据通道
Auto Research 是否形成一个可审稿的实证结论？ 研究问题、工件、对照、复跑、局限和成本账本 不把自动生成文本或局部指标当作科学结论
Long horizon 数小时工作能否持续、恢复且保持正确状态？ 持久状态、checkpoint、恢复记录、时间与失败轨迹 不把更多 token 或单次 demo 当成长期能力
Environment reward、动作、初态和 verifier 是否真的可控？ reset、权限、独立判题、证据轨与对抗回归 不能把容器、LLM judge 或截图单独当作 outcome
Synthetic training 学到的经验是否迁移到未见任务？ 有来源的实例/轨迹、过滤、训练配方与 held-out 改进 训练数据、teacher 与 evaluator 不能共享答案钥匙

这也给项目分工一个清楚的答案：Frontier Auto Research / ART 位于“研究工件”层；ClawBench V2 和 WebsiteBench 位于“环境”层；RewardHarness → RSIBench → OpenRSI 位于“候选改动与接纳”层；合成实例与 agentic RL 是跨层的训练层；long-horizon harness 贯穿所有需要多阶段工作的系统。

2. RSI：把“改进自己”降到可检查的接纳协议

新证据位 来源 最窄结论 对项目的含义
R1 Reward Hacking as Equilibrium 有限 evaluator 覆盖会结构性地留下未被优化的质量维度；这是理论建模主张 不能只增加一个总分；要枚举未覆盖维度与反例
R2 RewardHackingAgents evaluator tampering 与 train/test leakage 可以被设计为可观测的 benchmark outcome RSIBench 应把完整性失败与任务成功分开报告
R3 RSI survey “改什么”与“loop closure 程度”可把 self-refine、self-train、self-evaluate 和 research loop 分开 本项目应声明修改对象与人工/外部接纳边界
R4 Hack-Verifiable Environments 可在环境中植入可确定检测的 hacking 机会，而不是仅事后人工读轨迹 每个新 task family 应有反作弊 regression cases
R5 MetaSkill-Evolve frozen backbone 下可同时研究 task skill 与 improvement meta-skill 的演化 它是受限经验结果，不是对通用强 RSI 的证明
R6 GRASP skill library 的候选可用 balanced holdout 与 hard regression budget 接纳 acceptance gate、回归预算和被拒候选应成为一等工件

RSIBench / OpenRSI 的收束。 一个 RSI task 不应奖励“候选看起来更聪明”，而应验证：候选在声明 editable surface 中提出改动；development 过程记录所有尝试；independent acceptance 使用冻结且不可写的 evaluator；被接纳版本在密封的新任务上、同资源政策下不退化。至少分别报告 candidate_gain、acceptance_gain、heldout_gain、regression_rate 与 integrity_violations。只有最后两项同时成立，才有资格讨论“改进改进者”。

3. Auto Research：从检索、复现到新的可审稿证据包

新证据位 来源 最窄结论 对项目的含义
A1 SciAgentArena 真实科学场景可做成 interactive、stepwise-verifiable tasks；开放洞见仍显著困难 ART 要区分 workflow execution 与 novelty claim
A2 AutoResearchBench 深度定位论文和广泛收集满足条件的文献是可测的研究能力 literature evidence 应有 recall/precision 与来源链，不只给摘要
A3 ARA 论文中的 sources、methods、experiments、outputs 可抽为可重建 workflow graph ART 工件应显式记录依赖与结果生成路径
A4 SocSci-Repro-Bench 复现任务应区分 agent 失败与原始材料本身不可复现 每个 ART task 需有“材料可运行性”标签与失败归因
A5 EurekAgent permissions、artifacts、budget 和 human oversight 是科研 agent 环境的独立工程面 ART 需要声明权限、Git 工件、预算与人工干预点
A6 Meta-Agent Challenge 让 agent 在 sandbox 中开发另一个 agent 可成为带 held-out API 和 anti-hacking 控制的评测 “agent improves agent”须测最终 artifact，而不是只测 meta-agent 的叙述

ART 的收束。 Frontier Auto Research 不和 AutoLab 比“优化得更久”，不和 MLS-Bench 比“方法题更多”，也不和 EdgeBench 比“环境更长”。它的贡献单元是一个最小研究包：问题与前提、可执行的 baseline、允许改动面、实验图、强对照、负结果、复跑脚本、数据/代码版本、成本账本，以及由保留或新鲜设置支持的一条窄结论。它可以产出论文候选；是否有新颖性和领域价值仍由同行评审决定。

4. Long horizon：能力包含记忆、恢复、计划与安全退化

新证据位 来源 最窄结论 对项目的含义
L1 WildClawBench 原生 CLI harness、真实工具与 hybrid grading 会显著影响长期评测结果 报告 model、harness 与 evaluator 的组合，不只报模型名
L2 AgentLAB memory poisoning、objective drift 等攻击可在 long-horizon 环境中单独测量 long-horizon success 旁必须有安全失败率
L3 DeepPlanning 主动信息获取与全局时间/预算约束不同于局部 step reasoning ART 与 WebsiteBench 任务应包含全局约束而非只拆小步骤
L4 AMA-Bench agent memory 是机器生成的交互流，不能只用对话 QA 评测 state store 必须保存因果、目标和外部事实，而非仅相似检索
L5 VLAs-as-Tools 长程控制可拆为高层计划、局部工具和 progress feedback，并触发重规划 harness 应暴露可验证的子任务状态与 recovery event
L6 BCER Agent artifact binding 和 bounded local recovery 可使长链 workflow 的终态可追溯 每次恢复须关联输入、工具版本、输出和影响范围

Long-horizon harness 的验收。 每轮必须保留 task manifest、初态 hash、进度账本、checkpoint、工具/模型/环境版本、恢复原因、重试、人工接管和最后的 evaluator 记录。核心指标不是单一 pass rate，而是 outcome、recovery_success、state_reconstruction、unsafe_action_rate、cost_per_success 与 time_to_stable_artifact。这样才能区分“偶尔做完”与“可持续地工作”。

5. Environment：在训练前先让 reward 有一个可审计的出处

新证据位 来源 最窄结论 对项目的含义
E1 VeriEnv 可重建网站与内部 SDK 可把任务做成可执行、可验证的合成环境 self-hosted site 也要验证 cloned semantics 与未见站点迁移
E2 WebAgentGuard prompt injection detection 可与主 agent 解耦为独立 guard guard 也需测 false block、latency 与未见攻击样式
E3 EvoEnv 环境必须保持 solve–verify asymmetry，才能持续提供信息性 reward generator 不能让策略轻易以自然语言模拟或篡改 oracle
E4 BrowseSafe 真实 HTML 注入可改变 browser agent 的现实动作，而不是只影响文本 页面内容是非可信输入；特权与 judge 上下文不可暴露给页面
E5 Autonomous Evaluation for CUAs GUI 中的视觉 evaluator 可作为 noisy reward，但需显式做噪声校正 raw LLM/VLM judge 不能直接等同 ground truth
E6 MacArena 跨平台分布会改变 agent 排名；macOS-native task 与移植 task 不应混为一谈 WebsiteBench 需要按平台、UI 接口和分布分别报告
E7 OSWorld VM snapshot、执行式评测和多种 observation/action interface 可共同定义 computer-use environment screenshot 成功不能替代文件、cookie、服务端或业务 postcondition

ClawBench V2 / WebsiteBench 的收束。 环境协议应将五个平面写成版本化 contract：reset（完整 backend、账号、浏览器/VM 状态）、observe（DOM/CDP、a11y、截图或 terminal）、act（权限、egress 与被拦截的 commit surface）、verify（独立 judge 与 hidden rule）、evidence（append-only trace、artifact hash、网络/请求记录）。对每一个 reward，都至少放入“endpoint 对但 payload 错”“页面文本注入”“截图合理但服务端未提交”“trace 缺失/重排”四类反例。

6. 合成实例与基础模型训练：规模不是证据，外推才是

新证据位 来源 最窄结论 对项目的含义
S1 Trajectory Diversity Scaling 固定预算下 trajectory diversity 可比纯数量更有用 2K 目标要报告多样性、重复率和长尾覆盖，不只报行数
S2 RL Foundation Models synthetic MDP prior 可用于训练在 held-out tabular tasks 上 in-context 适应的模型 合成世界的先验必须公开；结果不自动迁移到 web/GUI agents
S3 EnvFactory 可执行 tool environment 与自然 multi-turn trajectory 可协同合成 环境/轨迹生成的任何成功结论都须保留 source、verifier 和 split
S4 ASTRA tool-call topology 可支持结构化轨迹合成和 rule-verifiable multi-turn RL trajectory-level reward 仍需用独立 outcome 审计 credit assignment
S5 SFT Memorizes, RL Generalizes 在研究的受控文本/视觉任务中，SFT 与 outcome-RL 的外推特性不同，且 SFT 可帮助 RL 稳定 这不是 browser RL 或所有数据分布上的算法胜负结论
S6 ACuRL 目标环境经验可驱动 curriculum task synthesis 与持续适应 target-environment 生成数据必须同最终 benchmark 隔离
S7 Foundation World Models foundation world model 需要把 specification、verification、calibration 和 test-time synthesis 当成同一研究议程 这是架构愿景，不是现成训练结果或安全保证

训练层的收束。 “2K synthetic instances”只能是一个可被推翻的规模假设。训练前必须有 provenance（源、许可、PII 审查、generator/checkpoint、模板和环境 image）；训练中有 SFT-only、RL-only、SFT+RL 与 frozen-base baselines；训练后只在按网站、workflow、账户模板、语义和 evaluator 规则隔离的任务上验收。报告 accepted/rejected trajectory、去重、difficulty/diversity、reward distribution、token、GPU、sandbox、API、wall-clock 与人工复核。否则“合成实例让基础模型变好”只是一句没有审计对象的宣传。

7. 现在可以做什么：一条由证据约束的项目路线

  1. 先建 ClawBench V2 / WebsiteBench 的环境核。 先冻结 reset、interception、evidence 和 independent verifier；用 anti-gaming regression suite 验证 reward 能代表 outcome。没有这一步，不进行大规模 RL。
  2. 再建合成实例管线。 只从与最终评测隔离的站点和 workflow 合成实例；记录每一条样本及其执行 verifier；先完成 SFT/RL/SFT+RL 的同预算 ablation。
  3. 用 long-horizon harness 把运行变成可恢复实验。 外部状态、checkpoint、重试和成本账本都进入 version control；并发只用于独立分支，合并前过同一 acceptance gate。
  4. 让 ART 对研究工件负责。 每个任务有问题、baseline、因果对照、复跑与失败结论；论文是可能产物，不是 reward。
  5. 最后才把 acceptance protocol 提升到 RSIBench / OpenRSI。 在同资源、独立 holdout 和不可写 evaluator 下，测试改善机制是否跨代保留收益。没有这一层，所有“RSI”表述都保持为受限 self-improvement。

这条路线使项目的差异可说清楚：我们不是另做一个“分数更高的 benchmark”，而是把 research outcome、环境控制、训练数据与 evaluator integrity 固定在同一个可审计系统中。EdgeBench 研究长程环境学习，AutoLab 研究固定预算的工件优化，MLS-Bench 研究方法泛化；我们的研究纲领补上它们之间通常被省略的证据链：候选能改什么、谁接纳、数据从哪里来、reward 指向什么、失败如何被保存，以及改进能否在未见条件下存活。

References

A Research-Agent Agenda: Five Threads, 32 New Evidence Slots

Jul 2026 · RSI / Auto Research / Long Horizon / Environments / Synthetic Training

The conclusion of this second research pass is not another list of benchmark names. It is a program organized around five distinct, rejectable artifacts: controlled self-improvement, reproducible research artifacts, recoverable long-horizon execution, auditable environments and evaluators, and synthetic training experience that transfers to unseen work. A credible research-agent program must state what each layer owns; a positive result in one layer cannot vouch for another.

Evidence status: these 32 new slots come from papers, public benchmarks, or official project materials, and many remain preprints. Each table states only the narrowest usable conclusion. ART, ClawBench V2, RSIBench / OpenRSI, and WebsiteBench remain research directions or working names here unless separately released publicly. This note reports no existing training score, accepted paper, or product status for them.

1. One Program, Five Non-Interchangeable Layers

Layer Question it answers Its actual output Constraint it places on the next layer
RSI Does the improvement process itself improve across generations? Versioned candidates, independent acceptance, rollback, and a cross-generation curve The candidate cannot write the acceptance gate or its evidence channel
Auto Research Does the system form a reviewable empirical conclusion? Research question, artifact, controls, reruns, limits, and cost ledger Generated prose or a local metric is not a scientific conclusion
Long horizon Can hours of work persist, recover, and retain the right state? Durable state, checkpoints, recovery records, time and failure traces More tokens or one demo is not long-horizon capability
Environment Are reward, action, initial state, and verification actually controlled? Reset, authority, independent judging, evidence trails, adversarial regressions A container, an LLM judge, or a screenshot alone is not an outcome
Synthetic training Does learned experience transfer to unseen work? Provenanced instances/trajectories, filtering, recipe, held-out gains Training data, teacher, and evaluator cannot share an answer key

This also answers the division of work. Frontier Auto Research / ART belongs to the research-artifact layer. ClawBench V2 and WebsiteBench belong to the environment layer. RewardHarness -> RSIBench -> OpenRSI belongs to the candidate-change and acceptance layer. Synthetic instances and agentic RL form a cross-cutting training layer. A long-horizon harness runs through every system that needs multi-stage work.

2. RSI: Reduce “Improving Itself” to an Inspectable Acceptance Protocol

New slot Source Narrow conclusion Project implication
R1 Reward Hacking as Equilibrium Finite evaluator coverage structurally leaves quality dimensions unoptimized; this is a theoretical modeling claim Do not add only one aggregate score; enumerate unmeasured dimensions and counterexamples
R2 RewardHackingAgents Evaluator tampering and train/test leakage can be engineered as observable benchmark outcomes RSIBench should report integrity failures separately from task success
R3 RSI survey What changes and how closed the loop is can separate self-refinement, self-training, self-evaluation, and research loops State the editable object and human/external acceptance boundary
R4 Hack-Verifiable Environments Environments can contain deterministically detectable hacking opportunities rather than relying only on post-hoc trace reading Every new task family needs anti-gaming regression cases
R5 MetaSkill-Evolve A frozen-backbone experiment can evolve task skills and an improvement meta-skill together It is bounded empirical evidence, not proof of general strong RSI
R6 GRASP Candidates for a skill library can be admitted with a balanced holdout and hard regression budget The acceptance gate, regression budget, and rejected candidates must be first-class artifacts

Convergence for RSIBench / OpenRSI. An RSI task should not reward a candidate for appearing smarter. It should verify that a candidate proposes changes inside a declared editable surface; development records every attempt; independent acceptance uses a frozen, unwritable evaluator; and accepted versions do not regress on sealed new tasks under an equal resource policy. Report candidate_gain, acceptance_gain, heldout_gain, regression_rate, and integrity_violations separately. Only the final two layers together justify discussing an “improver that improves.”

3. Auto Research: From Retrieval and Reproduction to a Reviewable Evidence Package

New slot Source Narrow conclusion Project implication
A1 SciAgentArena Real scientific settings can become interactive, stepwise-verifiable tasks; open insight remains difficult ART must separate workflow execution from a novelty claim
A2 AutoResearchBench Deep target-paper retrieval and broad literature collection are measurable research capabilities Literature evidence needs recall/precision and a source chain, not only a summary
A3 ARA Sources, methods, experiments, and outputs in papers can be extracted into a reconstructable workflow graph ART artifacts should expose dependencies and result-generation paths
A4 SocSci-Repro-Bench Reproduction tasks should distinguish agent failure from non-reproducible original materials Each ART task needs a material-runnability label and failure attribution
A5 EurekAgent Permissions, artifacts, budgets, and human oversight are independent engineering planes for research agents ART must declare authority, Git artifacts, budget, and human intervention points
A6 Meta-Agent Challenge An agent developing another agent in a sandbox can be evaluated through a held-out API and anti-hacking controls “An agent improves an agent” must evaluate the final artifact, not the meta-agent’s narrative

Convergence for ART. Frontier Auto Research should not compete with AutoLab by “optimizing longer,” with MLS-Bench by “having more method tasks,” or with EdgeBench by “having a longer environment.” Its unit of contribution is a minimum research package: problem and assumptions; runnable baseline; declared editable surface; experiment graph; strong controls; negative results; rerun script; data/code versions; cost ledger; and one narrow conclusion supported by held-out or fresh settings. It may produce a paper candidate; novelty and field value remain decisions of peer review.

4. Long Horizon: Capability Includes Memory, Recovery, Planning, and Safe Degradation

New slot Source Narrow conclusion Project implication
L1 WildClawBench Native CLI harnesses, real tools, and hybrid grading substantially change long-run evaluation Report the model-harness-evaluator configuration, not a model name alone
L2 AgentLAB Memory poisoning and objective drift can be measured separately in long-horizon environments Long-horizon success needs a safety-failure rate beside it
L3 DeepPlanning Active information gathering plus global time/budget constraints differs from local step reasoning ART and WebsiteBench tasks should include global constraints, not only decomposed steps
L4 AMA-Bench Agent memory is a machine-generated interaction stream, not just dialogue QA State stores need causality, goals, and external facts, not similarity retrieval alone
L5 VLAs-as-Tools Long-horizon control can combine high-level planning, bounded tools, progress feedback, and replanning The harness should expose verifiable subtask state and recovery events
L6 BCER Agent Artifact binding and bounded local recovery can make long chains traceable Every recovery should bind inputs, tool version, outputs, and blast radius

Long-horizon harness acceptance. Preserve a task manifest, initial-state hash, progress ledger, checkpoints, tool/model/environment versions, recovery causes, retries, human takeovers, and final evaluator record. The core metrics are not one pass rate but outcome, recovery_success, state_reconstruction, unsafe_action_rate, cost_per_success, and time_to_stable_artifact. This distinguishes an occasional completion from sustained work.

5. Environment: Before Training, Give Reward an Auditable Origin

New slot Source Narrow conclusion Project implication
E1 VeriEnv Recreated websites plus an internal SDK can yield executable, verifiable synthetic environments A self-hosted site still needs cloned-semantics checks and transfer to unseen sites
E2 WebAgentGuard Prompt-injection detection can be decoupled into a guard separate from the acting agent Evaluate false blocks, latency, and unseen attack styles too
E3 EvoEnv Environments need solve–verify asymmetry to keep rewards informative A generator must not let the policy simulate or tamper with its oracle in language
E4 BrowseSafe Realistic HTML injections can change browser-agent actions, not just text output Page content is untrusted input; it cannot see privileged or judge context
E5 Autonomous Evaluation for CUAs A GUI visual evaluator can be a noisy reward if that noise is explicitly corrected A raw LLM/VLM judge is not ground truth
E6 MacArena Cross-platform distributions can invert agent rankings; macOS-native and ported tasks differ WebsiteBench should report platform, UI interface, and distribution separately
E7 OSWorld VM snapshots, execution-based evaluation, and multiple observation/action interfaces can define computer-use environments Screenshot success does not replace file, cookie, server, or business postconditions

Convergence for ClawBench V2 / WebsiteBench. Write five planes as versioned contracts: reset (backend, identity, browser/VM state); observe (DOM/CDP, a11y, screenshot, or terminal); act (authority, egress, intercepted commit surface); verify (independent judge and hidden rules); and evidence (append-only traces, artifact hashes, network/request records). For every reward, include at least four counterexamples: right endpoint/wrong payload; injected page text; plausible screenshot/no server submission; and missing or reordered trace evidence.

6. Synthetic Instances and Foundation-Model Training: Scale Is Not Evidence, Extrapolation Is

New slot Source Narrow conclusion Project implication
S1 Trajectory Diversity Scaling Under a fixed budget, trajectory diversity can be more useful than raw quantity A 2K target must report diversity, duplication, and long-tail coverage, not only rows
S2 RL Foundation Models A synthetic MDP prior can train in-context adaptation on held-out tabular tasks Publish the synthetic-world prior; do not infer transfer to web/GUI agents
S3 EnvFactory Executable tool environments and natural multi-turn trajectories can be synthesized together Retain source, verifier, and split for every generation claim
S4 ASTRA Tool-call topology can support structured trajectory synthesis and rule-verifiable multi-turn RL Trajectory-level reward still needs independent outcome audits of credit assignment
S5 SFT Memorizes, RL Generalizes In its controlled text/vision studies, SFT and outcome RL differ in extrapolation, while SFT can stabilize later RL This is not a browser-RL result or an algorithm verdict for all distributions
S6 ACuRL Experience in a target environment can drive curriculum-task synthesis and continual adaptation Target-environment-generated data must remain isolated from final evaluation
S7 Foundation World Models A foundation-world-model agenda joins specifications, verification, calibration, and test-time synthesis It is an architectural vision, not a released training result or safety guarantee

Convergence for training. “2K synthetic instances” is only a falsifiable scale hypothesis. Before training, record provenance: source, license, PII review, generator/checkpoint, templates, and environment image. During training, compare SFT-only, RL-only, SFT+RL, and frozen-base baselines. After training, accept only on tasks split away by website, workflow, account template, semantics, and evaluator rules. Report accepted/rejected trajectories, deduplication, difficulty/diversity, reward distribution, tokens, GPUs, sandboxes, APIs, wall clock, and human review. Without that, “synthetic instances improved a foundation model” has no auditable object.

7. What to Build Now: an Evidence-Constrained Project Path

  1. Build the ClawBench V2 / WebsiteBench environment core first. Freeze reset, interception, evidence, and an independent verifier. Verify that reward tracks outcome through an anti-gaming regression suite. Do not scale RL before this exists.
  2. Then build the synthetic-instance pipeline. Synthesize only from sites and workflows isolated from final evaluation. Record every example and its execution verifier. Complete same-budget SFT/RL/SFT+RL ablations first.
  3. Use a long-horizon harness to turn runs into recoverable experiments. Put external state, checkpoints, retries, and the cost ledger in version control. Use parallelism only for independent branches; merge through the same acceptance gate.
  4. Make ART accountable to research artifacts. Every task has a question, baseline, causal controls, rerun, and failure conclusion. A paper is a possible artifact, never the reward.
  5. Only then elevate the acceptance protocol to RSIBench / OpenRSI. Under equal resource limits, an independent holdout, and an unwritable evaluator, test whether the improvement mechanism retains gains across generations. Without this layer, every RSI statement remains bounded self-improvement.

This makes the program’s differentiation clear. We are not building another “higher-score benchmark.” We are fixing research outcome, environment control, training data, and evaluator integrity inside one auditable system. EdgeBench studies long-horizon environment learning, AutoLab fixed-budget artifact optimization, and MLS-Bench method generalization. This agenda supplies the evidence chain often omitted between them: what a candidate may edit, who accepts it, where data originates, what reward denotes, how failures persist, and whether improvement survives unseen conditions.

References