1
00:00:00,340 --> 00:00:08,280
Imagine four analysis stages: import records, apply a required
transformation, build a chart, and export.

2
00:00:09,400 --> 00:00:14,760
The chart may look plausible even when the agent skipped the
transformation specified in the request.

3
00:00:15,460 --> 00:00:20,700
An export can therefore succeed as a software action while
delivering the wrong underlying result.

4
00:00:21,900 --> 00:00:26,660
The stages are connected by their input state, not simply by the
order of visible windows.

5
00:00:27,320 --> 00:00:31,680
This is a fictional short workflow, not one of the benchmark's
measured software tasks.

6
00:00:32,259 --> 00:00:38,220
It introduces the long-horizon continuity that Workflow-GYM asks
computer-use agents to maintain.

7
00:00:38,840 --> 00:00:47,200
Workflow-GYM evaluates complete computer-use workflows that
require professional software and domain-specific knowledge.

8
00:00:48,020 --> 00:00:55,020
Its pinned source contains three hundred thirty-eight tasks
across six domains and twenty-three finer subdomains.

9
00:00:55,860 --> 00:01:02,620
The tasks require thirty to one hundred ten annotated
operations, rather than a handful of isolated clicks.

10
00:01:03,460 --> 00:01:10,659
This creates dependencies between configuration, intermediate
state, and the final file or interface outcome.

11
00:01:11,580 --> 00:01:16,620
A locally reasonable action can still be wrong when the earlier
workflow state is incorrect.

12
00:01:17,240 --> 00:01:23,680
The benchmark therefore makes sustained execution part of what
it measures, alongside individual interface actions.

13
00:01:24,300 --> 00:01:32,300
The environment design separates reusable software-level virtual
machines from lightweight task-specific initialization.

14
00:01:33,540 --> 00:01:39,860
Fifty-six base virtual machines provide installed professional
software and its controlled version configuration.

15
00:01:40,760 --> 00:01:47,540
For a task, input files, templates, or settings are injected
through defined initialization procedures.

16
00:01:48,460 --> 00:01:54,520
The software starts ready for interaction, but the requested
task operations have not already been performed.

17
00:01:55,540 --> 00:02:02,140
Irrelevant login prerequisites are removed, allowing the
evaluation to focus on the specified professional workflow.

18
00:02:03,300 --> 00:02:09,380
In our diagram, these are generic workspaces, not credentials or
environments belonging to real individuals.

19
00:02:10,020 --> 00:02:17,040
Each task begins with a self-contained instruction describing
its objective, required software, and final outcome.

20
00:02:17,640 --> 00:02:22,540
The outcome may be a produced artifact or a specified final
graphical-interface state.

21
00:02:23,740 --> 00:02:30,280
Experts provide atomic procedures and execute them during
validation to check that the task can be completed.

22
00:02:31,040 --> 00:02:37,540
They revise problems in the environment, instructions, or
evaluation criteria when those prevent a valid solution.

23
00:02:38,440 --> 00:02:44,500
The main agent evaluation instead gives one initial instruction,
without step-by-step intermediate guidance.

24
00:02:45,180 --> 00:02:50,220
The expert procedure defines the task; it is not the guidance
supplied during main evaluation runs.

25
00:02:50,860 --> 00:02:57,440
Difficulty bands follow the length of the annotated procedure:
thirty to forty-four operations form the easy group.

26
00:02:58,220 --> 00:03:04,520
Forty-five to sixty operations form the medium group, and
sixty-one to one hundred ten form the hard group.

27
00:03:05,400 --> 00:03:10,920
These counts characterize task complexity, rather than
specifying the number of model interactions actually used.

28
00:03:11,960 --> 00:03:18,200
Source analysis describes stage omission, error propagation, and
objective drift across these extended workflows.

29
00:03:19,240 --> 00:03:24,220
Skipping a required stage can leave later actions operating on a
state that was never properly constructed.

30
00:03:25,160 --> 00:03:30,760
Our original transformation example shows that problem without
borrowing any real software or benchmark trajectory.

31
00:03:31,387 --> 00:03:37,547
The paper distinguishes workflow incompletion from reaching the
final stage with an incorrect result.

32
00:03:38,467 --> 00:03:43,827
An unfinished attempt may stop before export or leave the
software in an intermediate state.

33
00:03:44,767 --> 00:03:51,427
Another attempt can produce the expected file yet violate a
required configuration or omit a component.

34
00:03:52,887 --> 00:03:59,307
Artifact tasks check whether the produced files satisfy the
specified requirements rather than merely existing.

35
00:04:00,267 --> 00:04:07,387
Non-artifact tasks use final-interface evidence and predefined
judging criteria to assess the requested state.

36
00:04:08,507 --> 00:04:14,227
The evaluation target remains the end-to-end outcome, not just
the plausibility of individual clicks.

37
00:04:15,007 --> 00:04:20,467
Each model receives a single starting instruction and can use up
to four hundred interaction rounds.

38
00:04:21,727 --> 00:04:27,807
The source uses model-specific agent frameworks, rather than one
identical harness for every model.

39
00:04:29,227 --> 00:04:36,147
Every task is attempted three times independently, producing one
thousand fourteen trials across the full task set.

40
00:04:37,587 --> 00:04:44,487
Average Pass pools success across those trials, while Pass at
three reports the separate three-attempt measure.

41
00:04:46,047 --> 00:04:51,927
A greater chance of passing within repeated attempts is not the
same as reliable performance on each attempt.

42
00:04:53,167 --> 00:04:58,247
The following comparison keeps the same model row while
distinguishing those two reported metrics.

43
00:04:58,913 --> 00:05:05,573
For Gemini three point one Pro, the table reports Average Pass
of thirty point six seven percent.

44
00:05:06,213 --> 00:05:10,653
The same row reports Pass at three of forty-one point one two
percent.

45
00:05:11,473 --> 00:05:17,753
They summarize the same evaluation protocol through different
success measures; neither value should replace the other.

46
00:05:18,813 --> 00:05:25,833
Kimi K two point six has a different Pass at three value, so
pairing it with Gemini's average would mix models.

47
00:05:26,933 --> 00:05:33,153
These figures belong to the three hundred thirty-eight curated
workflows and their model-specific execution settings.

48
00:05:33,693 --> 00:05:39,413
They should not be interpreted as a general success guarantee
for arbitrary professional software tasks.

49
00:05:40,040 --> 00:05:46,740
Return to the original pipeline: the report depends on the
chart, and the chart depends on the transformation.

50
00:05:47,500 --> 00:05:54,000
A skipped stage can invalidate that chain while later software
actions still appear locally reasonable.

51
00:05:55,040 --> 00:06:01,760
Long-horizon computer use therefore requires maintaining the
intended state, not merely continuing to issue actions.

52
00:06:02,620 --> 00:06:08,520
Workflow-GYM tests continuity in controlled professional
software with defined end-to-end outcomes.

53
00:06:09,460 --> 00:06:15,440
The useful distinction is between reaching a final window and
producing the final state the user requested.

54
00:06:16,100 --> 00:06:22,000
A workflow succeeds when the important dependencies remain
correct all the way through the delivered outcome.