Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic Tasks in Real-World Professional Fields

Paper: Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic Tasks in Real-World Professional Fields (v4) https://arxiv.org/html/2606.11042v4

Workflow-GYM evaluates complete professional computer-use tasks from a controlled starting environment to a defined final outcome. The task unit is a workflow, with dependencies that can carry an early omission or error into later artifacts.

Each task supplies one complete instruction and a target artifact or GUI state. Software/version-specific base images and task initialization establish a reproducible workspace. Expert atomic procedures help validate the instruction, environment and assessment before the task enters the benchmark.

An original data-analysis example makes one dependency concrete. Imported values of 120, 80 and 50 are in tenths. A required conversion divides them by ten, giving 12, 8 and 5 for plotting and export. A skipped transform can preserve a tidy chart layout while changing what its numbers mean. The film’s values and operation are illustrative rather than a measured task trace.

Outcome inspection depends on the task type. Artifact tasks check required files; other tasks use final GUI screenshots and defined visual-model criteria. The source analysis distinguishes unfinished workflows from completed but incorrect outcomes, including omission, error propagation and objective drift.

The suite has 338 tasks in six domains and 23 subdomains, with a stated 30–110 expert-operation difficulty range. The main model comparison uses model-specific harnesses with one initial instruction, a shared 400-round cap and three independent trials.

AvgPass pools 1,014 trials. Pass@3 is a separate three-attempt metric. Gemini3.1Pro reports 30.67% AvgPass and 41.12 Pass@3; these definitions should accompany the numbers. The practical reading is to trace whether required transformations and conditions survive into the delivered artifact, while keeping performance claims tied to the evaluated tasks and protocol.

Pinned source versions: Paper, v4: https://arxiv.org/html/2606.11042v4

#ComputerUse #AIAgents #AIResearch