1
00:00:00,320 --> 00:00:06,780
Imagine an inventory request: prepare a workbook and a short PDF
summary from the supplied tables.

2
00:00:07,500 --> 00:00:12,860
The output looks tidy, yet the summary totals disagree with the
workbook's final values.

3
00:00:13,600 --> 00:00:18,720
A polished cover and several correct sheets cannot replace that
missing cross-file requirement.

4
00:00:19,640 --> 00:00:24,600
The user asked for usable files, rather than evidence that the
agent performed many steps.

5
00:00:25,580 --> 00:00:31,000
This fictional example contains no real organization, personal
data, or measured benchmark score.

6
00:00:31,880 --> 00:00:37,100
It introduces the difference between substantial progress and
meeting the final delivery conditions.

7
00:00:37,767 --> 00:00:43,927
StartupBench begins with AI-product workflows supported by
evidence of adoption or paid usage.

8
00:00:44,827 --> 00:00:51,167
Interviews recover the context, practical constraints, and
deliverables that users expect from those workflows.

9
00:00:52,347 --> 00:00:58,207
Market evidence motivates what to evaluate, but it does not
establish that an agent will succeed.

10
00:00:59,207 --> 00:01:05,927
Experts turn scenarios into reproducible tasks while preserving
the original objectives and expected deliverables.

11
00:01:06,807 --> 00:01:13,967
This connects evaluation to requested deliverables instead of
relying only on isolated researcher-designed subtasks.

12
00:01:15,027 --> 00:01:20,687
The resulting question is whether an artifact satisfies the
requirements that made the workflow useful.

13
00:01:21,093 --> 00:01:28,093
Each task combines a natural-language request, a workspace
containing inputs, and weighted evaluation rubrics.

14
00:01:28,913 --> 00:01:33,793
The request describes the output, while the workspace supplies
the information required to produce it.

15
00:01:34,713 --> 00:01:41,613
Rubrics separate requirements about functionality, structure,
formatting, and the quality expected within the domain.

16
00:01:42,373 --> 00:01:47,293
In the original example, one check inspects the workbook and
another compares its summary.

17
00:01:48,093 --> 00:01:54,013
Those conditions specify a deliverable more precisely than
asking the agent to make something professional.

18
00:01:54,753 --> 00:01:59,453
They also identify the evidence an evaluator should examine
after the agent finishes its attempt.

19
00:02:00,153 --> 00:02:06,033
The benchmark contains ninety-seven workflow tasks spread across
six top-level professional domains.

20
00:02:06,933 --> 00:02:14,653
Requested outputs include documents, spreadsheets, slides, PDFs,
images, and other specified work-product formats.

21
00:02:15,693 --> 00:02:21,133
Experts review the workspace, workflow, rubrics, and consistency
with the reference deliverable.

22
00:02:22,093 --> 00:02:27,853
Pilot runs reveal ambiguous instructions or missing resources
that require revision before a task is admitted.

23
00:02:28,913 --> 00:02:34,713
The process also selects challenging tasks, so results describe
this curated benchmark distribution.

24
00:02:35,493 --> 00:02:40,353
That task definition matters when relating a measured completion
rate to professional work more broadly.

25
00:02:40,980 --> 00:02:47,740
The evaluator uses extracted content and rendered pages while
retaining access to the original artifacts.

26
00:02:48,400 --> 00:02:54,360
Each rubric receives a separate judging session, producing a
binary decision and textual justification.

27
00:02:55,160 --> 00:03:01,020
This lets each check focus on one requirement instead of merging
everything into a general impression.

28
00:03:02,060 --> 00:03:07,180
Weighted aggregation produces the task score from the
requirements that the judge finds satisfied.

29
00:03:08,240 --> 00:03:14,680
For our workbook, a summary screenshot alone may not establish
that formulas and final totals agree.

30
00:03:15,300 --> 00:03:19,240
The original files may contain the evidence needed to test that
condition directly.

31
00:03:20,027 --> 00:03:26,207
Weighted scores give credit for satisfied requirements, even
when important requirements are still missing.

32
00:03:27,487 --> 00:03:33,327
The success metric applies another condition: a task run must
score at least ninety.

33
00:03:34,247 --> 00:03:40,167
That defines benchmark success, rather than perfect satisfaction
of every individual rubric item.

34
00:03:41,267 --> 00:03:47,907
Substantial average credit can therefore coexist with many
negative decisions about accepted final delivery.

35
00:03:49,467 --> 00:03:55,307
The source analysis identifies missing core conditions and
errors in verifying the final artifact.

36
00:03:56,387 --> 00:04:02,867
Our well-formatted workbook illustrates the gap when its
cross-file consistency condition remains unsatisfied.

37
00:04:03,553 --> 00:04:09,553
Nine evaluated models use the same Nanobot harness and tool
configuration in the main experiment.

38
00:04:10,393 --> 00:04:15,673
Each receives its task workspace and query, with a maximum of
two hundred interaction steps.

39
00:04:16,633 --> 00:04:21,753
Each model runs three times, and success is pooled over those
runs under the stated threshold.

40
00:04:22,813 --> 00:04:27,973
Average score instead summarizes weighted rubric credit, giving
a different view of the same attempts.

41
00:04:29,193 --> 00:04:35,633
The common harness supports comparison without making all
underlying architectures and model capabilities identical.

42
00:04:36,433 --> 00:04:40,793
Any numerical result should retain both its model row and its
metric definition.

43
00:04:41,347 --> 00:04:46,927
GPT five point six sol reports an average weighted score of
seventy-three point six one.

44
00:04:47,587 --> 00:04:53,647
The same model reports threshold success of thirty-one point two
seven percent over the pooled runs.

45
00:04:54,307 --> 00:05:00,667
That supports the approximate thirty-percent completion account,
rather than interpreting average score as pass rate.

46
00:05:01,547 --> 00:05:07,807
Kimi K three slightly leads mean score at seventy-three point
six seven but has lower threshold success.

47
00:05:08,387 --> 00:05:13,187
Its success is twenty-nine point five five percent, so the two
rankings need not match.

48
00:05:13,967 --> 00:05:19,287
These benchmark values do not specify a success probability for
an arbitrary workplace request.

49
00:05:19,867 --> 00:05:25,547
Return to the inventory example: the workbook and summary must
support the same requested purpose.

50
00:05:26,467 --> 00:05:32,827
Many completed steps can create convincing files while the
central consistency requirement remains unsatisfied.

51
00:05:33,787 --> 00:05:40,227
StartupBench asks whether market-grounded tasks produce
artifacts satisfying expert-defined acceptance conditions.

52
00:05:40,927 --> 00:05:46,407
Its scores preserve partial progress, while the success
threshold exposes the remaining delivery gap.

53
00:05:47,227 --> 00:05:52,307
An agent's confidence about its process should not replace
inspection of the actual final files.

54
00:05:52,987 --> 00:05:58,367
The useful question is whether the important requirements
survived all the way into the delivered work product.