1
00:00:00,360 --> 00:00:03,460
Imagine asking an agent to apply for a job using your resume.

2
00:00:04,019 --> 00:00:08,960
It finds the vacancy, extracts your details, uploads a file, and
completes several forms.

3
00:00:09,560 --> 00:00:14,600
But a finished-looking page cannot tell you whether the actual
submission contains the right information.

4
00:00:15,200 --> 00:00:19,120
ClawBench studies this kind of everyday task on live, production
websites.

5
00:00:19,600 --> 00:00:23,900
The central question is simple: can an agent carry your goal
through the entire workflow?

6
00:00:24,480 --> 00:00:28,160
We will follow an illustrative application, then see how the
benchmark checks the evidence.

7
00:00:28,733 --> 00:00:34,733
The local paper contains one hundred fifty-three tasks across
one hundred forty-four platforms and fifteen categories.

8
00:00:35,353 --> 00:00:41,513
These include purchases, reservations, applications, and other
workflows that would change information on a server.

9
00:00:42,333 --> 00:00:47,633
Each task has a user instruction, a starting address, and a
specified final submission request.

10
00:00:48,453 --> 00:00:53,633
Human annotators complete the task and record a reference
trajectory, including its submission payload.

11
00:00:54,433 --> 00:00:59,153
They filter out unavailable sites, paid subscriptions, and
geographically restricted services,

12
00:00:59,513 --> 00:01:00,713
and check reproducibility.

13
00:01:01,513 --> 00:01:07,733
For our example, the goal becomes specific: the right
application, the right resume, and correctly filled fields.

14
00:01:08,333 --> 00:01:11,253
A real submission could place an order or send an application.

15
00:01:11,773 --> 00:01:15,013
ClawBench therefore intercepts the designated final request.

16
00:01:15,493 --> 00:01:20,773
A human identifies its endpoint, request method, and required
payload fields during the reference run.

17
00:01:21,493 --> 00:01:27,573
The instrumentation captures the matching request body, blocks
it before transmission, and logs it for evaluation.

18
00:01:28,253 --> 00:01:33,333
Other page loads and dynamic requests pass through, preserving
ordinary interaction with the live website.

19
00:01:34,113 --> 00:01:39,033
In our example, the agent can reach Submit; the application
request is captured instead of sent.

20
00:01:39,693 --> 00:01:42,233
This boundary keeps the evidence of intended submission

21
00:01:42,553 --> 00:01:45,693
while preventing that annotated transaction from reaching the
server.

22
00:01:46,267 --> 00:01:51,387
The framework records five synchronized evidence streams, rather
than relying on one final screenshot.

23
00:01:52,087 --> 00:01:55,907
Session video and action screenshots show the visible page and
how it changes.

24
00:01:56,587 --> 00:02:03,067
HTTP traffic shows outgoing request data; browser actions record
clicks, typing, scrolling, and navigation.

25
00:02:03,847 --> 00:02:08,507
Agent messages preserve the framework's reasoning traces, tool
calls, and intermediate outputs.

26
00:02:09,327 --> 00:02:14,247
Human reference runs use the same recording infrastructure,
making comparison across the streams possible.

27
00:02:14,947 --> 00:02:19,427
Together, these records let a reviewer connect what was seen,
attempted, and submitted.

28
00:02:19,933 --> 00:02:25,633
A Claude Code sub-agent evaluates the task under a fixed rubric,
using both recorded trajectories.

29
00:02:26,273 --> 00:02:32,193
It aligns corresponding steps, checks required fields and
actions, and compares the terminal submission state.

30
00:02:33,013 --> 00:02:36,333
Consider our application: the uploaded resume is correct,

31
00:02:36,733 --> 00:02:39,673
but the outgoing phone-number field contains a different number.

32
00:02:40,313 --> 00:02:45,773
The payload exposes that mismatch even if the screenshot looked
complete or the agent claimed success.

33
00:02:46,733 --> 00:02:53,893
Interception alone is therefore insufficient. Required actions
and information must also satisfy the evaluation rules.

34
00:02:54,733 --> 00:02:59,793
The evaluator returns a binary verdict with structured
justification grounded in the recorded evidence.

35
00:03:00,400 --> 00:03:06,280
The rubric must also handle borderline cases, including blocked
submissions and equivalent alternative paths.

36
00:03:06,980 --> 00:03:10,660
The authors refine the prompt against calibration examples
carrying human verdicts.

37
00:03:11,200 --> 00:03:14,620
They evaluate the whole set, then collect disagreements with
those human labels.

38
00:03:15,280 --> 00:03:19,160
Each disagreement triggers diagnosis and targeted rule edits to
correct that case.

39
00:03:19,740 --> 00:03:24,600
Next, they retest every calibration example, exposing
regressions introduced by the local changes.

40
00:03:25,120 --> 00:03:30,340
The resulting prompt governs later evaluation: specific repairs,
checked across the whole calibration set.

41
00:03:30,900 --> 00:03:32,160
Now the measured outcome.

42
00:03:32,760 --> 00:03:37,680
The local experiment evaluates six models through the OpenClaw
framework, controlling a Chromium browser.

43
00:03:38,480 --> 00:03:42,840
Claude Sonnet four point six leads at thirty-three point three
percent task success.

44
00:03:43,500 --> 00:03:48,580
GLM-five reaches twenty-four point two percent; Gemini three
Flash reaches nineteen percent.

45
00:03:49,140 --> 00:03:55,300
Claude Haiku four point five scores eighteen point three
percent, while GPT-five point four reaches six point five.

46
00:03:56,020 --> 00:03:59,060
Gemini three point one Flash Lite scores three point three
percent.

47
00:03:59,820 --> 00:04:05,240
These are success rates for this paper's task set and execution
setup, with completion judged against human references.

48
00:04:06,007 --> 00:04:10,227
The recordings also explain where the runs break down, rather
than stopping at a leaderboard.

49
00:04:10,947 --> 00:04:17,447
Across nine hundred eighteen task-model runs, six hundred
nineteen are classified as blocked by anti-bot systems.

50
00:04:17,887 --> 00:04:22,387
That is sixty-seven point four percent of all runs, including
successful ones in the denominator.

51
00:04:23,167 --> 00:04:29,227
This makes live-site access a major measured obstacle in this
setup; the category does not isolate reasoning ability.

52
00:04:30,067 --> 00:04:35,547
Other recorded outcomes include policy refusals, website errors,
incomplete work, and repetitive loops.

53
00:04:36,287 --> 00:04:41,447
For developers, the useful question becomes: which step failed,
and what does its evidence actually show?

54
00:04:42,053 --> 00:04:49,073
ClawBench combines everyday tasks, targeted interception, human
demonstrations, and traceable evaluation.

55
00:04:49,953 --> 00:04:54,653
Our example separates a plausible application screen from a
correctly assembled submission.

56
00:04:55,333 --> 00:04:58,773
The final action is examined without sending the annotated
transaction.

57
00:04:59,893 --> 00:05:04,533
Reported success measures the model, browser framework, and live
environment together.

58
00:05:05,233 --> 00:05:09,093
The research direction is to make useful assistance observable,
step by step.

59
00:05:09,933 --> 00:05:14,113
Confidence comes from completed work, supported by evidence you
can inspect.