WEBVTT

1
00:00:00.320 --> 00:00:06.780
Imagine an inventory request: prepare a workbook and a short PDF
summary from the supplied tables.

2
00:00:07.500 --> 00:00:12.860
The output looks tidy, yet the summary totals disagree with the
workbook's final values.

3
00:00:13.600 --> 00:00:18.720
A polished cover and several correct sheets cannot replace that
missing cross-file requirement.

4
00:00:19.640 --> 00:00:24.600
The user asked for usable files, rather than evidence that the
agent performed many steps.

5
00:00:25.580 --> 00:00:31.000
This fictional example contains no real organization, personal
data, or measured benchmark score.

6
00:00:31.880 --> 00:00:37.100
It introduces the difference between substantial progress and
meeting the final delivery conditions.

7
00:00:37.767 --> 00:00:43.927
StartupBench begins with AI-product workflows supported by
evidence of adoption or paid usage.

8
00:00:44.827 --> 00:00:51.167
Interviews recover the context, practical constraints, and
deliverables that users expect from those workflows.

9
00:00:52.347 --> 00:00:58.207
Market evidence motivates what to evaluate, but it does not
establish that an agent will succeed.

10
00:00:59.207 --> 00:01:05.927
Experts turn scenarios into reproducible tasks while preserving
the original objectives and expected deliverables.

11
00:01:06.807 --> 00:01:13.967
This connects evaluation to requested deliverables instead of
relying only on isolated researcher-designed subtasks.

12
00:01:15.027 --> 00:01:20.687
The resulting question is whether an artifact satisfies the
requirements that made the workflow useful.

13
00:01:21.093 --> 00:01:28.093
Each task combines a natural-language request, a workspace
containing inputs, and weighted evaluation rubrics.

14
00:01:28.913 --> 00:01:33.793
The request describes the output, while the workspace supplies
the information required to produce it.

15
00:01:34.713 --> 00:01:41.613
Rubrics separate requirements about functionality, structure,
formatting, and the quality expected within the domain.

16
00:01:42.373 --> 00:01:47.293
In the original example, one check inspects the workbook and
another compares its summary.

17
00:01:48.093 --> 00:01:54.013
Those conditions specify a deliverable more precisely than
asking the agent to make something professional.

18
00:01:54.753 --> 00:01:59.453
They also identify the evidence an evaluator should examine
after the agent finishes its attempt.

19
00:02:00.153 --> 00:02:06.033
The benchmark contains ninety-seven workflow tasks spread across
six top-level professional domains.

20
00:02:06.933 --> 00:02:14.653
Requested outputs include documents, spreadsheets, slides, PDFs,
images, and other specified work-product formats.

21
00:02:15.693 --> 00:02:21.133
Experts review the workspace, workflow, rubrics, and consistency
with the reference deliverable.

22
00:02:22.093 --> 00:02:27.853
Pilot runs reveal ambiguous instructions or missing resources
that require revision before a task is admitted.

23
00:02:28.913 --> 00:02:34.713
The process also selects challenging tasks, so results describe
this curated benchmark distribution.

24
00:02:35.493 --> 00:02:40.353
That task definition matters when relating a measured completion
rate to professional work more broadly.

25
00:02:40.980 --> 00:02:47.740
The evaluator uses extracted content and rendered pages while
retaining access to the original artifacts.

26
00:02:48.400 --> 00:02:54.360
Each rubric receives a separate judging session, producing a
binary decision and textual justification.

27
00:02:55.160 --> 00:03:01.020
This lets each check focus on one requirement instead of merging
everything into a general impression.

28
00:03:02.060 --> 00:03:07.180
Weighted aggregation produces the task score from the
requirements that the judge finds satisfied.

29
00:03:08.240 --> 00:03:14.680
For our workbook, a summary screenshot alone may not establish
that formulas and final totals agree.

30
00:03:15.300 --> 00:03:19.240
The original files may contain the evidence needed to test that
condition directly.

31
00:03:20.027 --> 00:03:26.207
Weighted scores give credit for satisfied requirements, even
when important requirements are still missing.

32
00:03:27.487 --> 00:03:33.327
The success metric applies another condition: a task run must
score at least ninety.

33
00:03:34.247 --> 00:03:40.167
That defines benchmark success, rather than perfect satisfaction
of every individual rubric item.

34
00:03:41.267 --> 00:03:47.907
Substantial average credit can therefore coexist with many
negative decisions about accepted final delivery.

35
00:03:49.467 --> 00:03:55.307
The source analysis identifies missing core conditions and
errors in verifying the final artifact.

36
00:03:56.387 --> 00:04:02.867
Our well-formatted workbook illustrates the gap when its
cross-file consistency condition remains unsatisfied.

37
00:04:03.553 --> 00:04:09.553
Nine evaluated models use the same Nanobot harness and tool
configuration in the main experiment.

38
00:04:10.393 --> 00:04:15.673
Each receives its task workspace and query, with a maximum of
two hundred interaction steps.

39
00:04:16.633 --> 00:04:21.753
Each model runs three times, and success is pooled over those
runs under the stated threshold.

40
00:04:22.813 --> 00:04:27.973
Average score instead summarizes weighted rubric credit, giving
a different view of the same attempts.

41
00:04:29.193 --> 00:04:35.633
The common harness supports comparison without making all
underlying architectures and model capabilities identical.

42
00:04:36.433 --> 00:04:40.793
Any numerical result should retain both its model row and its
metric definition.

43
00:04:41.347 --> 00:04:46.927
GPT five point six sol reports an average weighted score of
seventy-three point six one.

44
00:04:47.587 --> 00:04:53.647
The same model reports threshold success of thirty-one point two
seven percent over the pooled runs.

45
00:04:54.307 --> 00:05:00.667
That supports the approximate thirty-percent completion account,
rather than interpreting average score as pass rate.

46
00:05:01.547 --> 00:05:07.807
Kimi K three slightly leads mean score at seventy-three point
six seven but has lower threshold success.

47
00:05:08.387 --> 00:05:13.187
Its success is twenty-nine point five five percent, so the two
rankings need not match.

48
00:05:13.967 --> 00:05:19.287
These benchmark values do not specify a success probability for
an arbitrary workplace request.

49
00:05:19.867 --> 00:05:25.547
Return to the inventory example: the workbook and summary must
support the same requested purpose.

50
00:05:26.467 --> 00:05:32.827
Many completed steps can create convincing files while the
central consistency requirement remains unsatisfied.

51
00:05:33.787 --> 00:05:40.227
StartupBench asks whether market-grounded tasks produce
artifacts satisfying expert-defined acceptance conditions.

52
00:05:40.927 --> 00:05:46.407
Its scores preserve partial progress, while the success
threshold exposes the remaining delivery gap.

53
00:05:47.227 --> 00:05:52.307
An agent's confidence about its process should not replace
inspection of the actual final files.

54
00:05:52.987 --> 00:05:58.367
The useful question is whether the important requirements
survived all the way into the delivered work product.