WEBVTT

1
00:00:00.360 --> 00:00:03.460
Imagine asking an agent to apply for a job using your resume.
想象一下，你请智能体根据你的简历申请一份工作。

2
00:00:04.019 --> 00:00:08.960
It finds the vacancy, extracts your details, uploads a file, and
completes several forms.
它找到职位，提取个人信息，上传文件，并填写几份表单。

3
00:00:09.560 --> 00:00:14.600
But a finished-looking page cannot tell you whether the actual
submission contains the right information.
但看似填写完成的页面，并不能说明实际提交的数据是否正确。

4
00:00:15.200 --> 00:00:19.120
ClawBench studies this kind of everyday task on live, production
websites.
ClawBench 在真实的生产网站上研究这类日常任务。

5
00:00:19.600 --> 00:00:23.900
The central question is simple: can an agent carry your goal
through the entire workflow?
核心问题很简单：智能体能否贯穿整个流程，完成你的目标？

6
00:00:24.480 --> 00:00:28.160
We will follow an illustrative application, then see how the
benchmark checks the evidence.
我们先跟着一个示意性的求职申请，再看看基准如何检查证据。

7
00:00:28.733 --> 00:00:34.733
The local paper contains one hundred fifty-three tasks across
one hundred forty-four platforms and fifteen categories.
这份本地论文包含 153 项任务，覆盖 144 个平台和 15 个类别。

8
00:00:35.353 --> 00:00:41.513
These include purchases, reservations, applications, and other
workflows that would change information on a server.
这些任务包括购买、预订、申请，以及其他会改变服务器信息的流程。

9
00:00:42.333 --> 00:00:47.633
Each task has a user instruction, a starting address, and a
specified final submission request.
每项任务都有用户指令、起始网址，以及指定的最终提交请求。

10
00:00:48.453 --> 00:00:53.633
Human annotators complete the task and record a reference
trajectory, including its submission payload.
人工标注者完成任务，并记录参考轨迹，其中包括提交载荷。

11
00:00:54.433 --> 00:00:59.153
They filter out unavailable sites, paid subscriptions, and
geographically restricted services,
他们排除不可用的网站、付费订阅服务和有地域限制的服务，

12
00:00:59.513 --> 00:01:00.713
and check reproducibility.
并检查可复现性。

13
00:01:01.513 --> 00:01:07.733
For our example, the goal becomes specific: the right
application, the right resume, and correctly filled fields.
在我们的示例中，目标变得具体：正确的申请、正确的简历，以及填写正确的字段。

14
00:01:08.333 --> 00:01:11.253
A real submission could place an order or send an application.
真实的提交可能会下订单或发送申请。

15
00:01:11.773 --> 00:01:15.013
ClawBench therefore intercepts the designated final request.
因此 ClawBench 会拦截指定的最终请求。

16
00:01:15.493 --> 00:01:20.773
A human identifies its endpoint, request method, and required
payload fields during the reference run.
人工标注者在参考执行中确定请求端点、请求方法，以及载荷中必须包含的字段。

17
00:01:21.493 --> 00:01:27.573
The instrumentation captures the matching request body, blocks
it before transmission, and logs it for evaluation.
检测系统捕获匹配的请求正文，在发送前将其阻止，并记录下来用于评估。

18
00:01:28.253 --> 00:01:33.333
Other page loads and dynamic requests pass through, preserving
ordinary interaction with the live website.
其他页面加载和动态请求照常通过，从而保留与真实网站的正常交互。

19
00:01:34.113 --> 00:01:39.033
In our example, the agent can reach Submit; the application
request is captured instead of sent.
在我们的示例中，智能体可以执行提交操作；申请请求会被捕获，而不会被发送。

20
00:01:39.693 --> 00:01:42.233
This boundary keeps the evidence of intended submission
这个边界保留了预期提交的证据，

21
00:01:42.553 --> 00:01:45.693
while preventing that annotated transaction from reaching the
server.
同时阻止那项已标注的交易到达服务器。

22
00:01:46.267 --> 00:01:51.387
The framework records five synchronized evidence streams, rather
than relying on one final screenshot.
该框架记录五条同步的证据流，而不是仅依赖一张最终截图。

23
00:01:52.087 --> 00:01:55.907
Session video and action screenshots show the visible page and
how it changes.
会话视频和操作截图显示可见页面，以及页面如何变化。

24
00:01:56.587 --> 00:02:03.067
HTTP traffic shows outgoing request data; browser actions record
clicks, typing, scrolling, and navigation.
HTTP 流量展示发出的请求数据；浏览器操作记录点击、输入、滚动和导航。

25
00:02:03.847 --> 00:02:08.507
Agent messages preserve the framework's reasoning traces, tool
calls, and intermediate outputs.
智能体消息保留框架记录的推理轨迹、工具调用和中间输出。

26
00:02:09.327 --> 00:02:14.247
Human reference runs use the same recording infrastructure,
making comparison across the streams possible.
人工参考执行使用同一套记录基础设施，因此可以对照这些证据流。

27
00:02:14.947 --> 00:02:19.427
Together, these records let a reviewer connect what was seen,
attempted, and submitted.
结合这些记录，审查者可以关联看到的内容、尝试的操作，以及提交的数据。

28
00:02:19.933 --> 00:02:25.633
A Claude Code sub-agent evaluates the task under a fixed rubric,
using both recorded trajectories.
Claude Code 子智能体依据固定的评估规则，使用两份已记录的轨迹来评估任务。

29
00:02:26.273 --> 00:02:32.193
It aligns corresponding steps, checks required fields and
actions, and compares the terminal submission state.
它对齐相应步骤，检查必需的字段和操作，并对照最终提交状态。

30
00:02:33.013 --> 00:02:36.333
Consider our application: the uploaded resume is correct,
看看我们的申请示例：上传的简历是正确的，

31
00:02:36.733 --> 00:02:39.673
but the outgoing phone-number field contains a different number.
但发出请求中的电话号码字段却是另一个号码。

32
00:02:40.313 --> 00:02:45.773
The payload exposes that mismatch even if the screenshot looked
complete or the agent claimed success.
即使截图看起来已填写完成，或者智能体声称成功，载荷仍会暴露这个不一致。

33
00:02:46.733 --> 00:02:53.893
Interception alone is therefore insufficient. Required actions
and information must also satisfy the evaluation rules.
因此，仅触发拦截还不够；必需的操作和信息也必须满足评估规则。

34
00:02:54.733 --> 00:02:59.793
The evaluator returns a binary verdict with structured
justification grounded in the recorded evidence.
评估器给出二元判定，并依据记录的证据提供结构化解释。

35
00:03:00.400 --> 00:03:06.280
The rubric must also handle borderline cases, including blocked
submissions and equivalent alternative paths.
评估规则还必须处理边界情况，包括提交被阻止，以及效果等价的替代路径。

36
00:03:06.980 --> 00:03:10.660
The authors refine the prompt against calibration examples
carrying human verdicts.
作者利用带有人工判定的校准示例来完善提示词。

37
00:03:11.200 --> 00:03:14.620
They evaluate the whole set, then collect disagreements with
those human labels.
他们评估整个集合，再收集与人工标签不一致的案例。

38
00:03:15.280 --> 00:03:19.160
Each disagreement triggers diagnosis and targeted rule edits to
correct that case.
每个不一致案例都会触发诊断，以及针对该案例的规则修改。

39
00:03:19.740 --> 00:03:24.600
Next, they retest every calibration example, exposing
regressions introduced by the local changes.
随后，他们重新测试所有校准示例，找出局部修改造成的退步。

40
00:03:25.120 --> 00:03:30.340
The resulting prompt governs later evaluation: specific repairs,
checked across the whole calibration set.
最终得到的提示词用于后续评估：修复具体问题，再用整个校准集合检查一致性。

41
00:03:30.900 --> 00:03:32.160
Now the measured outcome.
现在来看测量结果。

42
00:03:32.760 --> 00:03:37.680
The local experiment evaluates six models through the OpenClaw
framework, controlling a Chromium browser.
该论文通过 OpenClaw 框架评估六个模型，由它们控制 Chromium 浏览器。

43
00:03:38.480 --> 00:03:42.840
Claude Sonnet four point six leads at thirty-three point three
percent task success.
Claude Sonnet 4.6 以 33.3% 的任务成功率领先。

44
00:03:43.500 --> 00:03:48.580
GLM-five reaches twenty-four point two percent; Gemini three
Flash reaches nineteen percent.
GLM-5 达到 24.2%；Gemini 3 Flash 达到 19.0%。

45
00:03:49.140 --> 00:03:55.300
Claude Haiku four point five scores eighteen point three
percent, while GPT-five point four reaches six point five.
Claude Haiku 4.5 得分为 18.3%，而 GPT-5.4 达到 6.5%。

46
00:03:56.020 --> 00:03:59.060
Gemini three point one Flash Lite scores three point three
percent.
Gemini 3.1 Flash Lite 得分为 3.3%。

47
00:03:59.820 --> 00:04:05.240
These are success rates for this paper's task set and execution
setup, with completion judged against human references.
这些成功率对应论文中的任务集和执行配置，完成情况通过人工参考轨迹来判定。

48
00:04:06.007 --> 00:04:10.227
The recordings also explain where the runs break down, rather
than stopping at a leaderboard.
这些记录还能解释执行在哪一步中断，而不只是给出排行榜。

49
00:04:10.947 --> 00:04:17.447
Across nine hundred eighteen task-model runs, six hundred
nineteen are classified as blocked by anti-bot systems.
在 918 次任务与模型的组合执行中，619 次被归类为遭反机器人系统阻止。

50
00:04:17.887 --> 00:04:22.387
That is sixty-seven point four percent of all runs, including
successful ones in the denominator.
这占所有执行的 67.4%；分母也包括成功的执行。

51
00:04:23.167 --> 00:04:29.227
This makes live-site access a major measured obstacle in this
setup; the category does not isolate reasoning ability.
这说明在该配置下，真实网站的访问障碍是主要的实测困难；这个类别并不能单独衡量推理能力。

52
00:04:30.067 --> 00:04:35.547
Other recorded outcomes include policy refusals, website errors,
incomplete work, and repetitive loops.
其他记录的结果包括因安全政策而拒绝、网站错误、任务未完成，以及反复循环。

53
00:04:36.287 --> 00:04:41.447
For developers, the useful question becomes: which step failed,
and what does its evidence actually show?
对开发者来说，更有用的问题变成了：哪一步失败了，这一步的证据究竟说明了什么？

54
00:04:42.053 --> 00:04:49.073
ClawBench combines everyday tasks, targeted interception, human
demonstrations, and traceable evaluation.
ClawBench 结合了日常任务、定向拦截、人工示范，以及可追溯的评估。

55
00:04:49.953 --> 00:04:54.653
Our example separates a plausible application screen from a
correctly assembled submission.
我们的示例区分了看起来合理的申请页面，以及正确组织的提交数据。

56
00:04:55.333 --> 00:04:58.773
The final action is examined without sending the annotated
transaction.
最终操作会被检查，而那项已标注的交易不会被发送。

57
00:04:59.893 --> 00:05:04.533
Reported success measures the model, browser framework, and live
environment together.
报告的成功率共同反映了模型、浏览器框架，以及真实环境的表现。

58
00:05:05.233 --> 00:05:09.093
The research direction is to make useful assistance observable,
step by step.
这一研究方向，是让有用的辅助能力能够逐步被观察和检验。

59
00:05:09.933 --> 00:05:14.113
Confidence comes from completed work, supported by evidence you
can inspect.
信心来自完成的工作，以及可以检查的证据。