WEBVTT

1
00:00:00.320 --> 00:00:03.760
Suppose a research agent writes a comparison of two recycling
approaches,

2
00:00:03.980 --> 00:00:07.360
covering collection, costs, contamination, and reported
outcomes.

3
00:00:07.960 --> 00:00:09.220
The report may look polished

4
00:00:09.260 --> 00:00:14.480
while omitting a required comparison, drifting into a tangent,
or citing the wrong source page.

5
00:00:15.300 --> 00:00:19.300
Dr. Bench evaluates long-form research reports through several
distinct dimensions

6
00:00:19.340 --> 00:00:21.020
rather than a single short-answer check.

7
00:00:21.620 --> 00:00:26.940
Its framework combines semantic quality, topical focus, and
alignment with expert-designated sources.

8
00:00:27.600 --> 00:00:32.840
We will use this original recycling illustration to explain
those measurements, then read the reported score comparison.

9
00:00:33.340 --> 00:00:39.300
The key is to keep the questions separate: did it do the task,
stay on topic, and use the expected source evidence?

10
00:00:39.867 --> 00:00:45.247
The benchmark contains two hundred fourteen expert-curated tasks
across ten broad domains.

11
00:00:45.927 --> 00:00:50.927
Each query asks for a structured report and comes with a
manually constructed reference bundle.

12
00:00:51.547 --> 00:00:56.547
Five modules organize that bundle: task-specific rubrics,
general-report rubrics,

13
00:00:57.067 --> 00:01:00.527
source links, focus anchors, and deviation keywords.

14
00:01:01.287 --> 00:01:05.467
The query-specific criteria represent what this particular
report should accomplish.

15
00:01:06.047 --> 00:01:10.527
The other modules ask how well the report is constructed, which
sources it uses,

16
00:01:10.987 --> 00:01:13.327
and how closely it follows the intended topic.

17
00:01:14.127 --> 00:01:20.567
The construction pipeline includes machine auditing and multiple
manual reviews of rubrics, keywords, and source links.

18
00:01:21.180 --> 00:01:26.400
For our recycling example, a task rubric might require an
explicit comparison of costs

19
00:01:26.940 --> 00:01:29.320
and an explanation of why the approaches differ.

20
00:01:30.060 --> 00:01:31.680
Those are requirements for this question.

21
00:01:32.680 --> 00:01:39.220
General-report rubrics separately assess structure, clarity,
citation quality, and analytical rigor.

22
00:01:40.060 --> 00:01:43.760
A well-formatted report can therefore still miss the requested
analysis,

23
00:01:44.260 --> 00:01:46.580
while a relevant answer can be poorly organized.

24
00:01:47.600 --> 00:01:52.780
The Quality metric normalizes the two rubric scores and combines
them with weighted averaging.

25
00:01:53.540 --> 00:01:58.600
In the reported experiment, query-specific and general-report
quality receive equal weight.

26
00:01:59.360 --> 00:02:04.520
This produces a quality signal about content and expression, not
a task-pass percentage.

27
00:02:05.280 --> 00:02:10.760
Each task has five expert-defined focus-anchor terms and five
focus-deviation terms.

28
00:02:11.600 --> 00:02:18.300
In the recycling illustration, anchors could identify
collection, sorting, contamination, costs, and outcomes.

29
00:02:18.960 --> 00:02:23.400
A passage about unrelated vacation planning would be a tangent
to this hypothetical comparison.

30
00:02:24.400 --> 00:02:28.380
SemanticDrift measures both omitted anchors and the use of
deviation terms.

31
00:02:29.000 --> 00:02:33.140
It combines scaled keyword frequency with language-model
assessments of semantic relevance,

32
00:02:33.520 --> 00:02:35.120
rather than counting occurrences alone.

33
00:02:36.080 --> 00:02:42.040
Lower drift means stronger measured topical focus; the
integrated score uses the complementary focus factor.

34
00:02:42.593 --> 00:02:48.573
Experts provide trustworthy-source links to authoritative
primary pages containing information needed for the task.

35
00:02:49.353 --> 00:02:53.433
The source metric checks exact link matches and matches to the
same hostname.

36
00:02:54.093 --> 00:02:59.213
For our example, citing the designated report page and citing
another page from the same institution

37
00:02:59.273 --> 00:03:00.153
are different matches.

38
00:03:00.873 --> 00:03:05.013
The two match types have different weights and create a bounded
source-alignment boost.

39
00:03:05.793 --> 00:03:11.793
This checks alignment with an expert source list. A matching
domain does not prove that every cited claim is supported.

40
00:03:12.513 --> 00:03:17.773
Query-specific content rubrics and source-list matching
therefore play complementary roles in the report assessment.

41
00:03:18.393 --> 00:03:24.553
The integrated score multiplies Quality, the focus factor, and
the source boost, with the paper's score scaling.

42
00:03:25.093 --> 00:03:31.033
A strong quality signal can be reduced by topical drift, while
expected source coverage increases the source factor.

43
00:03:31.793 --> 00:03:33.973
The experiment compares thirteen systems,

44
00:03:34.252 --> 00:03:38.553
including dedicated deep-research agents and reasoning models
equipped with web search.

45
00:03:39.313 --> 00:03:43.393
The reported component columns and integrated column are
different aggregated measurements.

46
00:03:44.033 --> 00:03:46.393
We use the integrated scores as reported,

47
00:03:46.473 --> 00:03:50.273
rather than multiplying their component-column averages to
reconstruct the leaderboard.

48
00:03:50.993 --> 00:03:55.313
Those values are score points. They are not percentages of tasks
successfully completed.

49
00:03:55.960 --> 00:04:00.600
In Table One, Qwen Deep Research has about thirty-four point six
five integrated points.

50
00:04:01.260 --> 00:04:04.060
Sonar Deep Research follows at about thirty-three point four
seven,

51
00:04:04.540 --> 00:04:07.600
and o-three Deep Research at about thirty-two point nine zero.

52
00:04:08.220 --> 00:04:12.400
Kimi K-two has the highest mean Quality value, zero point six
seven zero seven,

53
00:04:12.820 --> 00:04:14.420
but does not lead the integrated ranking.

54
00:04:14.880 --> 00:04:17.780
Its integrated score is about thirty-two point zero seven
points.

55
00:04:18.500 --> 00:04:22.680
The comparison illustrates why task quality, measured focus, and
source alignment

56
00:04:22.720 --> 00:04:24.540
should remain visible as separate components.

57
00:04:25.320 --> 00:04:28.120
These are the paper's historical systems and scoring
configuration,

58
00:04:28.620 --> 00:04:30.800
not a current universal ranking of research assistants.

59
00:04:31.527 --> 00:04:38.327
Dr. Bench makes long-report evaluation concrete through
task-specific reference bundles and explicit scoring dimensions.

60
00:04:39.027 --> 00:04:45.247
The recycling illustration separates requested analysis from
presentation, topical focus, and source choice.

61
00:04:45.967 --> 00:04:51.627
A readable report still needs its required content checked. A
source match still needs the associated claims examined.

62
00:04:52.427 --> 00:04:57.767
The integrated score summarizes several signals, while the
component views help explain what that summary contains.

63
00:04:58.747 --> 00:05:01.887
The useful assessment asks whether the report answers this
question,

64
00:05:02.427 --> 00:05:05.207
stays with this topic, and organizes relevant evidence.

65
00:05:05.927 --> 00:05:10.187
That turns an impressive-looking document into a research output
whose quality can be inspected.