WEBVTT

1
00:00:00.320 --> 00:00:04.300
Imagine two agents using the same language model. One checks its
work,
想象两个使用同一语言模型的智能体。一个会检查工作，

2
00:00:04.500 --> 00:00:08.960
recovers from errors, and keeps useful context; the other stops
too early.
从错误中恢复，并保留有用的上下文；另一个却过早停止。

3
00:00:09.660 --> 00:00:14.740
Their surrounding software can change what the model
accomplishes. That software is the agent harness.
外围软件能够改变模型完成任务的能力，这套软件就是智能体的执行框架。

4
00:00:15.480 --> 00:00:21.360
HarnessDev asks whether a language model can build this
infrastructure, then improve the runnable system it created.
HarnessDev 研究语言模型能否构建这些基础设施，并进一步改进自己创建的可运行系统。

5
00:00:22.040 --> 00:00:27.240
The submitted object is persistent code, reused across tasks,
rather than one answer to one question.
提交对象是跨任务复用的持久代码，而不是对某个问题的一次回答。

6
00:00:27.840 --> 00:00:33.720
This changes the unit of evaluation: engineering the execution
system becomes the capability under test.
这改变了评估单位：设计和实现执行系统本身，成为被测试的能力。

7
00:00:34.540 --> 00:00:39.840
We will follow a completion-check example, then separate
development feedback from the later held-out test.
我们将跟着一个完成检查的例子，再区分开发反馈与之后的留出测试。

8
00:00:40.380 --> 00:00:47.240
Creation begins with a deliberately weak seed that can read
configuration, expose tools, and write the required logs.
创建阶段从一个刻意弱化的初始框架开始，它能够读取配置、提供工具，并写出所需日志。

9
00:00:47.720 --> 00:00:55.560
It has no agent loop, task decomposition, verification, recovery
logic, or stopping rule. Unmodified, it scores zero.
它没有智能体循环、任务分解、验证、恢复逻辑或停止规则；不修改时，它得分为零。

10
00:00:56.240 --> 00:01:01.680
The creator receives a task-family specification, constraints,
and one to three development cases.
创建模型获得任务族说明、约束，以及一到三个开发案例。

11
00:01:02.200 --> 00:01:06.740
It must connect the available primitives into a system that
actually attempts the downstream work.
它必须把现有基础组件连接成一个真正尝试执行下游任务的系统。

12
00:01:07.320 --> 00:01:11.620
Hidden evaluation tasks and the human implementation remain
unavailable during development.
在开发期间，隐藏评估任务和人工实现都不可见。

13
00:01:12.399 --> 00:01:18.500
A finished harness is then frozen before it is evaluated. The
benchmark measures what the added execution logic enables.
随后，完成的框架会在评估前冻结；基准衡量新增执行逻辑带来的能力。

14
00:01:19.167 --> 00:01:25.867
The creator writes the harness. The executor is the model that
runs inside the frozen harness on downstream tasks.
创建模型编写框架；执行模型则在冻结后的框架内部运行，处理下游任务。

15
00:01:26.627 --> 00:01:31.767
Keeping these roles separate lets the paper examine whether a
design depends on its original runtime model.
区分这两个角色，让论文能够检查某个设计是否依赖最初的运行时模型。

16
00:01:32.647 --> 00:01:36.887
Creation covers six creator models, four domains, and five
benchmarks,
创建评估覆盖六个创建模型、四个领域和五个基准，

17
00:01:37.087 --> 00:01:40.047
totaling two thousand two hundred seven unique instances.
共计 2,207 个不同实例。

18
00:01:40.947 --> 00:01:45.747
Generated harnesses show uneven performance: writing and
machine-learning experimentation
生成的框架表现不均衡：写作和机器学习实验

19
00:01:45.807 --> 00:01:49.647
are stronger than code and search relative to selected
references.
相对于选定参考系统，比代码与搜索表现更强。

20
00:01:50.327 --> 00:01:55.467
Those reference comparisons are system-level comparisons, and
some reference scores come from external reports.
这些参考对比属于系统层面的对比，部分参考分数来自外部报告。

21
00:01:56.287 --> 00:02:01.527
Changing the executor can also change the rankings, exposing
compatibility as part of harness quality.
更换执行模型也会改变排名，说明兼容性是框架质量的一部分。

22
00:02:02.233 --> 00:02:05.753
One evolution trajectory provides a concrete diagnostic case.
一条演化轨迹提供了具体的诊断案例。

23
00:02:06.453 --> 00:02:12.733
Opus finds that ninety-nine of one hundred runs report success,
but only forty-eight pass.
Opus 发现，一百次执行中有九十九次报告成功，但只有四十八次通过。

24
00:02:13.373 --> 00:02:18.233
It traces the gap to premature completion and adds a completion
check to the harness.
它把差距追溯到过早结束，并在框架中加入完成检查。

25
00:02:18.973 --> 00:02:23.913
Picture a gate before the final answer: has the requested
artifact been produced,
可以想象最终回答前的一道关卡：要求的产物是否已生成，

26
00:02:24.273 --> 00:02:26.573
and is there evidence that the task is complete?
是否有证据证明任务完成？

27
00:02:27.173 --> 00:02:32.353
That question belongs to the execution logic surrounding the
model, rather than a change to its weights.
这个问题属于模型外围的执行逻辑，而不是模型权重的变化。

28
00:02:33.113 --> 00:02:38.793
The paper identifies this gate as a positive example with
execution-path and case-level evidence.
论文把这道关卡视为一个具有执行路径和案例证据的正面例子。

29
00:02:39.413 --> 00:02:45.413
The lesson is specific: useful feedback exposes a failure that
an executable change can address.
这个启示很具体：有用的反馈揭示出可由可执行修改解决的故障。

30
00:02:46.073 --> 00:02:51.633
Evolution starts from the creator's own frozen code harness and
repeatedly evaluates revised commits.
演化从创建模型自己冻结后的代码框架开始，并反复评估修改后的提交版本。

31
00:02:52.293 --> 00:02:57.093
Visible feedback uses one hundred SWE-Pro tasks and eighty-nine
Terminal-Bench tasks.
可见反馈使用一百个 SWE-Pro 和八十九个 Terminal-Bench 任务。

32
00:02:57.633 --> 00:03:00.333
An official candidate must finish both evaluation legs.
正式候选版本必须完成这两部分评估。

33
00:03:01.273 --> 00:03:07.053
After development ends, every official version is tested on six
hundred thirty disjoint SWE-Pro instances.
开发结束后，每个正式版本都会在六百三十个不重叠的 SWE-Pro 实例上测试。

34
00:03:07.653 --> 00:03:12.093
These scores are never returned to the creator and cannot
influence its edits or final version choice.
这些分数不会反馈给创建模型，也不能影响其修改或最终版本选择。

35
00:03:12.833 --> 00:03:17.813
The feedback score averages two benchmark percentages; the
held-out score uses only SWE-Pro.
反馈分数对两个基准的百分比分数求平均；留出分数则只使用 SWE-Pro。

36
00:03:18.533 --> 00:03:23.753
A development gain and a held-out gain therefore answer
different questions. They must remain separate on the chart.
因此，开发阶段增益和留出增益回答的是不同问题，图表必须把二者区分开。

37
00:03:24.327 --> 00:03:28.167
Consider the GPT-five point five lineage executed by fixed
Gemini.
来看由固定 Gemini 执行的 GPT-5.5 谱系。

38
00:03:28.667 --> 00:03:32.787
Its feedback pair rises from fifty-six point six to fifty-nine
point one.
它的反馈配对分数从五十六点六升至五十九点一。

39
00:03:33.487 --> 00:03:40.627
On the held-out SWE-Pro set, the corresponding scores fall from
forty-two point two two to thirty-one point nine zero.
在留出 SWE-Pro 集合上，对应分数却从四十二点二二降至三十一点九零。

40
00:03:41.367 --> 00:03:46.707
The selected revision improved the visible pair while reducing
performance on that separate held-out set.
选定的修改提升了可见配对分数，却降低了在独立留出集合上的表现。

41
00:03:47.267 --> 00:03:52.087
This is one trajectory under one executor, not an average effect
attributed to that creator model.
这是一种执行模型下的一条轨迹，而不是归因于该创建模型的平均效应。

42
00:03:52.827 --> 00:03:56.726
Across self-runtime declarations, all five improve held-out
performance;
在使用各自运行时模型的最终版本中，五个都提升了留出表现；

43
00:03:57.167 --> 00:04:00.667
under fixed Gemini, only Opus improves and the other three
regress.
但固定 Gemini 时，只有 Opus 提升，其他三个都退步。

44
00:04:01.687 --> 00:04:06.467
Executor compatibility and generalization therefore belong in
the test of an evolved harness.
因此，执行模型兼容性和泛化表现，都应纳入演化框架的测试。

45
00:04:07.047 --> 00:04:11.687
The nine evolution lineages produce sixty-four adjacent official
version switches.
九条演化谱系一共产生六十四次相邻正式版本切换。

46
00:04:12.207 --> 00:04:17.587
Feedback and held-out scores move in the same direction on
thirty-four of those switches, about fifty-three percent.
其中三十四次切换的反馈和留出分数朝同一方向变化，约占百分之五十三。

47
00:04:18.167 --> 00:04:23.047
Only two of nine declared final versions are also the best
held-out versions within their lineages.
九个声明的最终版本中，只有两个也是各自谱系里留出表现最好的版本。

48
00:04:23.527 --> 00:04:29.367
Repeated runs of the same commit can vary, and many reported
gains remain inside the measured noise band.
同一提交版本的重复执行会有波动，许多报告的增益仍处于测得的噪声范围内。

49
00:04:29.907 --> 00:04:35.347
Selecting the highest visible score is therefore an unreliable
substitute for checking unseen tasks.
因此，选择最高可见分数，并不能可靠地替代对未见任务的检查。

50
00:04:35.927 --> 00:04:41.287
The benchmark treats version selection as part of development,
rather than assuming every edit is an improvement.
基准把版本选择视为开发的一部分，而不是假设每次编辑都代表改进。

51
00:04:41.740 --> 00:04:47.800
HarnessDev makes a model's surrounding execution system an
inspectable artifact that can persist across tasks.
HarnessDev 把模型外围的执行系统变成一个可检查、可跨任务持续使用的产物。

52
00:04:48.440 --> 00:04:54.400
Creation asks whether the system can be built; Evolution asks
whether feedback can improve the existing artifact.
创建阶段检查系统能否被构建；演化阶段检查反馈能否改进现有产物。

53
00:04:55.220 --> 00:04:58.220
The completion-gate example shows a constructive local repair.
完成关卡示例展示了建设性的局部修复。

54
00:04:58.720 --> 00:05:02.200
The held-out comparisons ask whether repairs travel beyond their
feedback set.
留出对比检查这些修复能否超越反馈集合发挥作用。

55
00:05:03.040 --> 00:05:07.860
Code must run, preserve useful behavior, and remain compatible
with the model that executes it.
代码必须能够运行，保留有用行为，并与执行它的模型保持兼容。

56
00:05:08.680 --> 00:05:13.740
A convincing self-developed agent needs evidence about both its
software and its downstream performance.
一个有说服力的自开发智能体，需要同时提供软件和下游表现的证据。

57
00:05:14.340 --> 00:05:20.780
The important result is an engineering task we can measure:
build, freeze, run, inspect, and revise.
重要的成果是一个能够测量的工程任务：构建、冻结、运行、检查，再修改。