WEBVTT

1
00:00:00.340 --> 00:00:08.280
Imagine four analysis stages: import records, apply a required
transformation, build a chart, and export.

2
00:00:09.400 --> 00:00:14.760
The chart may look plausible even when the agent skipped the
transformation specified in the request.

3
00:00:15.460 --> 00:00:20.700
An export can therefore succeed as a software action while
delivering the wrong underlying result.

4
00:00:21.900 --> 00:00:26.660
The stages are connected by their input state, not simply by the
order of visible windows.

5
00:00:27.320 --> 00:00:31.680
This is a fictional short workflow, not one of the benchmark's
measured software tasks.

6
00:00:32.259 --> 00:00:38.220
It introduces the long-horizon continuity that Workflow-GYM asks
computer-use agents to maintain.

7
00:00:38.840 --> 00:00:47.200
Workflow-GYM evaluates complete computer-use workflows that
require professional software and domain-specific knowledge.

8
00:00:48.020 --> 00:00:55.020
Its pinned source contains three hundred thirty-eight tasks
across six domains and twenty-three finer subdomains.

9
00:00:55.860 --> 00:01:02.620
The tasks require thirty to one hundred ten annotated
operations, rather than a handful of isolated clicks.

10
00:01:03.460 --> 00:01:10.659
This creates dependencies between configuration, intermediate
state, and the final file or interface outcome.

11
00:01:11.580 --> 00:01:16.620
A locally reasonable action can still be wrong when the earlier
workflow state is incorrect.

12
00:01:17.240 --> 00:01:23.680
The benchmark therefore makes sustained execution part of what
it measures, alongside individual interface actions.

13
00:01:24.300 --> 00:01:32.300
The environment design separates reusable software-level virtual
machines from lightweight task-specific initialization.

14
00:01:33.540 --> 00:01:39.860
Fifty-six base virtual machines provide installed professional
software and its controlled version configuration.

15
00:01:40.760 --> 00:01:47.540
For a task, input files, templates, or settings are injected
through defined initialization procedures.

16
00:01:48.460 --> 00:01:54.520
The software starts ready for interaction, but the requested
task operations have not already been performed.

17
00:01:55.540 --> 00:02:02.140
Irrelevant login prerequisites are removed, allowing the
evaluation to focus on the specified professional workflow.

18
00:02:03.300 --> 00:02:09.380
In our diagram, these are generic workspaces, not credentials or
environments belonging to real individuals.

19
00:02:10.020 --> 00:02:17.040
Each task begins with a self-contained instruction describing
its objective, required software, and final outcome.

20
00:02:17.640 --> 00:02:22.540
The outcome may be a produced artifact or a specified final
graphical-interface state.

21
00:02:23.740 --> 00:02:30.280
Experts provide atomic procedures and execute them during
validation to check that the task can be completed.

22
00:02:31.040 --> 00:02:37.540
They revise problems in the environment, instructions, or
evaluation criteria when those prevent a valid solution.

23
00:02:38.440 --> 00:02:44.500
The main agent evaluation instead gives one initial instruction,
without step-by-step intermediate guidance.

24
00:02:45.180 --> 00:02:50.220
The expert procedure defines the task; it is not the guidance
supplied during main evaluation runs.

25
00:02:50.860 --> 00:02:57.440
Difficulty bands follow the length of the annotated procedure:
thirty to forty-four operations form the easy group.

26
00:02:58.220 --> 00:03:04.520
Forty-five to sixty operations form the medium group, and
sixty-one to one hundred ten form the hard group.

27
00:03:05.400 --> 00:03:10.920
These counts characterize task complexity, rather than
specifying the number of model interactions actually used.

28
00:03:11.960 --> 00:03:18.200
Source analysis describes stage omission, error propagation, and
objective drift across these extended workflows.

29
00:03:19.240 --> 00:03:24.220
Skipping a required stage can leave later actions operating on a
state that was never properly constructed.

30
00:03:25.160 --> 00:03:30.760
Our original transformation example shows that problem without
borrowing any real software or benchmark trajectory.

31
00:03:31.387 --> 00:03:37.547
The paper distinguishes workflow incompletion from reaching the
final stage with an incorrect result.

32
00:03:38.467 --> 00:03:43.827
An unfinished attempt may stop before export or leave the
software in an intermediate state.

33
00:03:44.767 --> 00:03:51.427
Another attempt can produce the expected file yet violate a
required configuration or omit a component.

34
00:03:52.887 --> 00:03:59.307
Artifact tasks check whether the produced files satisfy the
specified requirements rather than merely existing.

35
00:04:00.267 --> 00:04:07.387
Non-artifact tasks use final-interface evidence and predefined
judging criteria to assess the requested state.

36
00:04:08.507 --> 00:04:14.227
The evaluation target remains the end-to-end outcome, not just
the plausibility of individual clicks.

37
00:04:15.007 --> 00:04:20.467
Each model receives a single starting instruction and can use up
to four hundred interaction rounds.

38
00:04:21.727 --> 00:04:27.807
The source uses model-specific agent frameworks, rather than one
identical harness for every model.

39
00:04:29.227 --> 00:04:36.147
Every task is attempted three times independently, producing one
thousand fourteen trials across the full task set.

40
00:04:37.587 --> 00:04:44.487
Average Pass pools success across those trials, while Pass at
three reports the separate three-attempt measure.

41
00:04:46.047 --> 00:04:51.927
A greater chance of passing within repeated attempts is not the
same as reliable performance on each attempt.

42
00:04:53.167 --> 00:04:58.247
The following comparison keeps the same model row while
distinguishing those two reported metrics.

43
00:04:58.913 --> 00:05:05.573
For Gemini three point one Pro, the table reports Average Pass
of thirty point six seven percent.

44
00:05:06.213 --> 00:05:10.653
The same row reports Pass at three of forty-one point one two
percent.

45
00:05:11.473 --> 00:05:17.753
They summarize the same evaluation protocol through different
success measures; neither value should replace the other.

46
00:05:18.813 --> 00:05:25.833
Kimi K two point six has a different Pass at three value, so
pairing it with Gemini's average would mix models.

47
00:05:26.933 --> 00:05:33.153
These figures belong to the three hundred thirty-eight curated
workflows and their model-specific execution settings.

48
00:05:33.693 --> 00:05:39.413
They should not be interpreted as a general success guarantee
for arbitrary professional software tasks.

49
00:05:40.040 --> 00:05:46.740
Return to the original pipeline: the report depends on the
chart, and the chart depends on the transformation.

50
00:05:47.500 --> 00:05:54.000
A skipped stage can invalidate that chain while later software
actions still appear locally reasonable.

51
00:05:55.040 --> 00:06:01.760
Long-horizon computer use therefore requires maintaining the
intended state, not merely continuing to issue actions.

52
00:06:02.620 --> 00:06:08.520
Workflow-GYM tests continuity in controlled professional
software with defined end-to-end outcomes.

53
00:06:09.460 --> 00:06:15.440
The useful distinction is between reaching a final window and
producing the final state the user requested.

54
00:06:16.100 --> 00:06:22.000
A workflow succeeds when the important dependencies remain
correct all the way through the delivered outcome.