ClawBench: Can AI Agents Complete Everyday Online Tasks?

Paper: ClawBench: Can AI Agents Complete Everyday Online Tasks? (v2) https://arxiv.org/abs/2604.08523v2

This explainer follows the mechanism before interpreting its reported results.

An online form can look complete while its actual submission still misses the user’s request. ClawBench turns that distinction into an evaluation question: can an agent finish an everyday task, and can the execution evidence establish that it did? Our application example is an original schematic, not a real person’s record.

The benchmark contains 153 tasks across 144 live platforms and 15 categories. A task specifies an instruction, starting URL and annotated final submission request. Human reference trajectories establish a concrete completion target. This is a live-environment evaluation, rather than a claim about isolated reasoning ability.

Interception has a deliberately precise scope. The system matches the designated endpoint, method and required payload fields, captures that terminal request, and blocks it before transmission. Ordinary page and dynamic requests continue. The rule should not be read as a universal safeguard against every financial or consequential submission.

Five synchronized streams record the execution: session video, screenshots, HTTP traffic, agent messages and browser actions. A fixed-rubric evaluator compares the agent and reference traces, checking required fields, actions or an equivalent terminal state. Its output is a binary verdict with structured justification. Calibration against human verdicts leads to targeted rubric repairs and regression checks across the calibration examples.

The six tested models have success rates of 33.3% for Sonnet 4.6, 24.2% for GLM-5, 19.0% for Gemini 3 Flash, 18.3% for Haiku 4.5, 6.5% for GPT-5.4, and 3.3% for Gemini 3.1 Flash-Lite. They run through the reported OpenClaw and Chromium setup.

Across 918 task-model runs, 619 were blocked by anti-bot mechanisms: 67.4% of all runs, not just failures. That denominator matters. The results measure a model interacting through a framework with a live environment. The central contribution is making completion claims traceable to a specified target and observable evidence.