Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to Reports

Paper: Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to Reports (v2) https://arxiv.org/html/2510.02190v2

This explainer follows the mechanism before interpreting its reported results.

Suppose a research agent compares two recycling approaches. The report may read smoothly while omitting contamination, drifting into an unrelated discussion, or citing a relevant-looking site instead of the intended page. Our original schematic makes these separate checks visible; it is not an actual benchmark report.

Dr. Bench contains 214 expert-curated tasks across ten domains. Each task has a reference bundle: query-specific rubrics, general report rubrics, trustworthy source links, five focus anchors and five deviation terms. This bundle lets the evaluation ask more than whether the prose is polished.

Quality combines normalized task-specific and general rubric scores. The reported experiment gives them equal weights. Topical focus uses SemanticDrift, which considers omitted anchors, deviation-term frequency and semantic relevance; the focus factor is one minus that drift.

Source alignment is a separate signal. TBoost is bounded and incorporates weighted exact-page and hostname matches against reference links. A hostname match is broader than a page match, and neither alone proves that every sentence is supported. It is useful to preserve that distinction when interpreting a citation-heavy report.

The integrated metric multiplies quality, the focus factor and TBoost within each report under the paper’s scaling. Its aggregate is therefore not reconstructed by multiplying the three displayed column averages. Different reports can combine the dimensions differently.

The historical experiment compares 13 systems: five deep-research agents, one advanced agent and seven search-enabled reasoning models. Qwen Deep Research scores 34.6480 integrated points, followed by Sonar Deep Research at 33.4668, o3 Deep Research at 32.9004, and Kimi K2 at 32.0651. Kimi K2 has the highest quality score, 0.6707, but does not lead the integrated ranking.

The paper gives a practical reading habit: inspect task coverage, focus and source alignment independently, then understand how the chosen combination changes the final comparison.