StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows

Paper: StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows (v1) https://arxiv.org/html/2608.17800v1

StartupBench: judge the files against the work’s purpose

A work product should be checked against the purpose that made it useful. StartupBench connects market-grounded workflows to explicit artifact requirements, then separates partial credit from threshold-defined success.

StartupBench begins with workflows supported by adoption or paid-usage evidence. Interviews recover objectives, constraints and expected outputs; experts reconstruct reproducible tasks. Market validation motivates what to evaluate. It does not prove that a tested agent can complete the work.

Each task combines a request, workspace inputs and weighted rubrics. The benchmark includes 97 tasks across six professional domains, with outputs such as documents, spreadsheets, slides and other specified formats. Expert review and pilots check specifications before admission. The resulting distribution is curated for challenge rather than a random sample of all workplace requests.

Evaluation examines extracted content, rendered pages and original artifacts. Each rubric receives a separate binary judgment and justification. Weighted aggregation credits satisfied requirements. Threshold success asks another question: did the run score at least 90? That is a defined benchmark threshold, not perfect satisfaction of every criterion.

Nine models share the main Nanobot harness and tools, a 200-interaction-step cap and three independent runs. GPT-5.6-sol reports 73.61 average weighted points and 31.27% pooled threshold success. Kimi K3 reports 73.67 points but 29.55% success. The slight ordering reversal shows why average credit is not completion rate.

The useful check is whether important acceptance conditions survived into the delivered files. A summary screenshot may miss formula or cross-file inconsistencies. StartupBench keeps partial progress visible while making the remaining delivery gap explicit under its task definitions.

Watch the companion English animation with external English, Simplified Chinese or bilingual captions.

Paper, exact arXiv v1: https://arxiv.org/html/2608.17800v1

#AIAgents #LLMEvaluation #AIResearch