1
00:00:00,360 --> 00:00:03,420
Imagine someone asking: help me plan a team dinner.

2
00:00:04,019 --> 00:00:07,140
The request sounds simple, until you remember the background.

3
00:00:07,640 --> 00:00:12,460
Suppose their team includes a vegetarian colleague, and one
member needs step-free access.

4
00:00:12,920 --> 00:00:16,700
An attractive restaurant list could still miss what makes the
plan useful.

5
00:00:17,500 --> 00:00:21,080
This is an illustrative request, rather than an example of a
real participant.

6
00:00:22,080 --> 00:00:27,220
xDailyBench studies this difference between following the words
and satisfying the task.

7
00:00:27,833 --> 00:00:32,533
The benchmark starts from requests people have used AI for, or
genuinely intended to complete.

8
00:00:33,093 --> 00:00:38,013
Each task supplies an instruction, optional workspace materials,
and relevant user context.

9
00:00:38,693 --> 00:00:44,673
Some requirements are explicit. Others reflect basic
expectations, such as accuracy or usable formatting.

10
00:00:45,373 --> 00:00:49,073
A third group depends on the particular user and the context
supplied with the task.

11
00:00:49,713 --> 00:00:53,833
The agent needs to connect those pieces, rather than invent a
background it was never given.

12
00:00:54,553 --> 00:00:58,513
The final collection includes two hundred forty-eight tasks
across fifty-one scenarios.

13
00:00:59,127 --> 00:01:04,967
An open-ended task can have several good answers. The rubric
defines what those answers must accomplish.

14
00:01:05,507 --> 00:01:11,507
For the dinner illustration, a plan might satisfy dietary needs,
access needs, and a practical schedule.

15
00:01:12,047 --> 00:01:16,307
These are separate criteria, so a judge can inspect each
requirement individually.

16
00:01:17,087 --> 00:01:20,627
Each criterion receives a binary judgement: met or not met.

17
00:01:21,087 --> 00:01:25,427
The criteria carry weights, reflecting their relative importance
in completing the task.

18
00:01:26,127 --> 00:01:31,567
The resulting score can credit useful progress without
pretending the whole task was completed perfectly.

19
00:01:32,133 --> 00:01:37,753
The paper uses assisted screening, then manual cross-review, to
refine the tasks and their criteria.

20
00:01:38,393 --> 00:01:44,413
Reviewers check whether each criterion is atomic, clear, and
suitable for a binary judgement.

21
00:01:45,193 --> 00:01:49,133
A pilot compares automatic and human judgements on the same
outputs and rubrics.

22
00:01:49,893 --> 00:01:53,633
Criteria that miss the required agreement threshold are reviewed
and revised.

23
00:01:54,333 --> 00:01:59,513
The final experiments compare eleven models with matched
instructions, tools, and execution budgets.

24
00:02:00,293 --> 00:02:05,493
The important object is the full response or deliverable,
assessed against the task requirements.

25
00:02:06,173 --> 00:02:11,193
In the reported results, two models tie at a task-level score of
seventy-five point six percent.

26
00:02:11,753 --> 00:02:17,433
That score averages task scores. It is not the percentage of
tasks completed without any missing criterion.

27
00:02:18,473 --> 00:02:22,093
Now inspect one model, Seed two point one Evolving, at the
rubric level.

28
00:02:22,733 --> 00:02:28,273
Its explicit criteria score is eighty-one percent; its implicit
criteria score is sixty-five percent.

29
00:02:28,793 --> 00:02:34,253
Those are pooled criterion scores, a sixteen-percentage-point
gap within this reported comparison.

30
00:02:34,853 --> 00:02:39,293
Looking only at the overall number would hide which part of
understanding the task needs improvement.

31
00:02:39,860 --> 00:02:46,240
Return to the dinner plan. A polished answer can still overlook
a need that was present in the supplied context.

32
00:02:47,080 --> 00:02:52,940
xDailyBench makes those requirements explicit in evaluation,
even when they were implicit in the request.

33
00:02:53,760 --> 00:03:00,600
Its contribution is a way to ask whether an assistant connects
instructions, background, and a usable final result.

34
00:03:01,260 --> 00:03:06,840
The lesson is to read the whole task before deciding that the
visible words have been satisfied.