1 00:00:00,320 --> 00:00:03,760 Suppose a research agent writes a comparison of two recycling approaches, 2 00:00:03,980 --> 00:00:07,360 covering collection, costs, contamination, and reported outcomes. 3 00:00:07,960 --> 00:00:09,220 The report may look polished 4 00:00:09,260 --> 00:00:14,480 while omitting a required comparison, drifting into a tangent, or citing the wrong source page. 5 00:00:15,300 --> 00:00:19,300 Dr. Bench evaluates long-form research reports through several distinct dimensions 6 00:00:19,340 --> 00:00:21,020 rather than a single short-answer check. 7 00:00:21,620 --> 00:00:26,940 Its framework combines semantic quality, topical focus, and alignment with expert-designated sources. 8 00:00:27,600 --> 00:00:32,840 We will use this original recycling illustration to explain those measurements, then read the reported score comparison. 9 00:00:33,340 --> 00:00:39,300 The key is to keep the questions separate: did it do the task, stay on topic, and use the expected source evidence? 10 00:00:39,867 --> 00:00:45,247 The benchmark contains two hundred fourteen expert-curated tasks across ten broad domains. 11 00:00:45,927 --> 00:00:50,927 Each query asks for a structured report and comes with a manually constructed reference bundle. 12 00:00:51,547 --> 00:00:56,547 Five modules organize that bundle: task-specific rubrics, general-report rubrics, 13 00:00:57,067 --> 00:01:00,527 source links, focus anchors, and deviation keywords. 14 00:01:01,287 --> 00:01:05,467 The query-specific criteria represent what this particular report should accomplish. 15 00:01:06,047 --> 00:01:10,527 The other modules ask how well the report is constructed, which sources it uses, 16 00:01:10,987 --> 00:01:13,327 and how closely it follows the intended topic. 17 00:01:14,127 --> 00:01:20,567 The construction pipeline includes machine auditing and multiple manual reviews of rubrics, keywords, and source links. 18 00:01:21,180 --> 00:01:26,400 For our recycling example, a task rubric might require an explicit comparison of costs 19 00:01:26,940 --> 00:01:29,320 and an explanation of why the approaches differ. 20 00:01:30,060 --> 00:01:31,680 Those are requirements for this question. 21 00:01:32,680 --> 00:01:39,220 General-report rubrics separately assess structure, clarity, citation quality, and analytical rigor. 22 00:01:40,060 --> 00:01:43,760 A well-formatted report can therefore still miss the requested analysis, 23 00:01:44,260 --> 00:01:46,580 while a relevant answer can be poorly organized. 24 00:01:47,600 --> 00:01:52,780 The Quality metric normalizes the two rubric scores and combines them with weighted averaging. 25 00:01:53,540 --> 00:01:58,600 In the reported experiment, query-specific and general-report quality receive equal weight. 26 00:01:59,360 --> 00:02:04,520 This produces a quality signal about content and expression, not a task-pass percentage. 27 00:02:05,280 --> 00:02:10,760 Each task has five expert-defined focus-anchor terms and five focus-deviation terms. 28 00:02:11,600 --> 00:02:18,300 In the recycling illustration, anchors could identify collection, sorting, contamination, costs, and outcomes. 29 00:02:18,960 --> 00:02:23,400 A passage about unrelated vacation planning would be a tangent to this hypothetical comparison. 30 00:02:24,400 --> 00:02:28,380 SemanticDrift measures both omitted anchors and the use of deviation terms. 31 00:02:29,000 --> 00:02:33,140 It combines scaled keyword frequency with language-model assessments of semantic relevance, 32 00:02:33,520 --> 00:02:35,120 rather than counting occurrences alone. 33 00:02:36,080 --> 00:02:42,040 Lower drift means stronger measured topical focus; the integrated score uses the complementary focus factor. 34 00:02:42,593 --> 00:02:48,573 Experts provide trustworthy-source links to authoritative primary pages containing information needed for the task. 35 00:02:49,353 --> 00:02:53,433 The source metric checks exact link matches and matches to the same hostname. 36 00:02:54,093 --> 00:02:59,213 For our example, citing the designated report page and citing another page from the same institution 37 00:02:59,273 --> 00:03:00,153 are different matches. 38 00:03:00,873 --> 00:03:05,013 The two match types have different weights and create a bounded source-alignment boost. 39 00:03:05,793 --> 00:03:11,793 This checks alignment with an expert source list. A matching domain does not prove that every cited claim is supported. 40 00:03:12,513 --> 00:03:17,773 Query-specific content rubrics and source-list matching therefore play complementary roles in the report assessment. 41 00:03:18,393 --> 00:03:24,553 The integrated score multiplies Quality, the focus factor, and the source boost, with the paper's score scaling. 42 00:03:25,093 --> 00:03:31,033 A strong quality signal can be reduced by topical drift, while expected source coverage increases the source factor. 43 00:03:31,793 --> 00:03:33,973 The experiment compares thirteen systems, 44 00:03:34,252 --> 00:03:38,553 including dedicated deep-research agents and reasoning models equipped with web search. 45 00:03:39,313 --> 00:03:43,393 The reported component columns and integrated column are different aggregated measurements. 46 00:03:44,033 --> 00:03:46,393 We use the integrated scores as reported, 47 00:03:46,473 --> 00:03:50,273 rather than multiplying their component-column averages to reconstruct the leaderboard. 48 00:03:50,993 --> 00:03:55,313 Those values are score points. They are not percentages of tasks successfully completed. 49 00:03:55,960 --> 00:04:00,600 In Table One, Qwen Deep Research has about thirty-four point six five integrated points. 50 00:04:01,260 --> 00:04:04,060 Sonar Deep Research follows at about thirty-three point four seven, 51 00:04:04,540 --> 00:04:07,600 and o-three Deep Research at about thirty-two point nine zero. 52 00:04:08,220 --> 00:04:12,400 Kimi K-two has the highest mean Quality value, zero point six seven zero seven, 53 00:04:12,820 --> 00:04:14,420 but does not lead the integrated ranking. 54 00:04:14,880 --> 00:04:17,780 Its integrated score is about thirty-two point zero seven points. 55 00:04:18,500 --> 00:04:22,680 The comparison illustrates why task quality, measured focus, and source alignment 56 00:04:22,720 --> 00:04:24,540 should remain visible as separate components. 57 00:04:25,320 --> 00:04:28,120 These are the paper's historical systems and scoring configuration, 58 00:04:28,620 --> 00:04:30,800 not a current universal ranking of research assistants. 59 00:04:31,527 --> 00:04:38,327 Dr. Bench makes long-report evaluation concrete through task-specific reference bundles and explicit scoring dimensions. 60 00:04:39,027 --> 00:04:45,247 The recycling illustration separates requested analysis from presentation, topical focus, and source choice. 61 00:04:45,967 --> 00:04:51,627 A readable report still needs its required content checked. A source match still needs the associated claims examined. 62 00:04:52,427 --> 00:04:57,767 The integrated score summarizes several signals, while the component views help explain what that summary contains. 63 00:04:58,747 --> 00:05:01,887 The useful assessment asks whether the report answers this question, 64 00:05:02,427 --> 00:05:05,207 stays with this topic, and organizes relevant evidence. 65 00:05:05,927 --> 00:05:10,187 That turns an impressive-looking document into a research output whose quality can be inspected.