WEBVTT

1
00:00:00.360 --> 00:00:03.420
Imagine someone asking: help me plan a team dinner.

2
00:00:04.019 --> 00:00:07.140
The request sounds simple, until you remember the background.

3
00:00:07.640 --> 00:00:12.460
Suppose their team includes a vegetarian colleague, and one
member needs step-free access.

4
00:00:12.920 --> 00:00:16.700
An attractive restaurant list could still miss what makes the
plan useful.

5
00:00:17.500 --> 00:00:21.080
This is an illustrative request, rather than an example of a
real participant.

6
00:00:22.080 --> 00:00:27.220
xDailyBench studies this difference between following the words
and satisfying the task.

7
00:00:27.833 --> 00:00:32.533
The benchmark starts from requests people have used AI for, or
genuinely intended to complete.

8
00:00:33.093 --> 00:00:38.013
Each task supplies an instruction, optional workspace materials,
and relevant user context.

9
00:00:38.693 --> 00:00:44.673
Some requirements are explicit. Others reflect basic
expectations, such as accuracy or usable formatting.

10
00:00:45.373 --> 00:00:49.073
A third group depends on the particular user and the context
supplied with the task.

11
00:00:49.713 --> 00:00:53.833
The agent needs to connect those pieces, rather than invent a
background it was never given.

12
00:00:54.553 --> 00:00:58.513
The final collection includes two hundred forty-eight tasks
across fifty-one scenarios.

13
00:00:59.127 --> 00:01:04.967
An open-ended task can have several good answers. The rubric
defines what those answers must accomplish.

14
00:01:05.507 --> 00:01:11.507
For the dinner illustration, a plan might satisfy dietary needs,
access needs, and a practical schedule.

15
00:01:12.047 --> 00:01:16.307
These are separate criteria, so a judge can inspect each
requirement individually.

16
00:01:17.087 --> 00:01:20.627
Each criterion receives a binary judgement: met or not met.

17
00:01:21.087 --> 00:01:25.427
The criteria carry weights, reflecting their relative importance
in completing the task.

18
00:01:26.127 --> 00:01:31.567
The resulting score can credit useful progress without
pretending the whole task was completed perfectly.

19
00:01:32.133 --> 00:01:37.753
The paper uses assisted screening, then manual cross-review, to
refine the tasks and their criteria.

20
00:01:38.393 --> 00:01:44.413
Reviewers check whether each criterion is atomic, clear, and
suitable for a binary judgement.

21
00:01:45.193 --> 00:01:49.133
A pilot compares automatic and human judgements on the same
outputs and rubrics.

22
00:01:49.893 --> 00:01:53.633
Criteria that miss the required agreement threshold are reviewed
and revised.

23
00:01:54.333 --> 00:01:59.513
The final experiments compare eleven models with matched
instructions, tools, and execution budgets.

24
00:02:00.293 --> 00:02:05.493
The important object is the full response or deliverable,
assessed against the task requirements.

25
00:02:06.173 --> 00:02:11.193
In the reported results, two models tie at a task-level score of
seventy-five point six percent.

26
00:02:11.753 --> 00:02:17.433
That score averages task scores. It is not the percentage of
tasks completed without any missing criterion.

27
00:02:18.473 --> 00:02:22.093
Now inspect one model, Seed two point one Evolving, at the
rubric level.

28
00:02:22.733 --> 00:02:28.273
Its explicit criteria score is eighty-one percent; its implicit
criteria score is sixty-five percent.

29
00:02:28.793 --> 00:02:34.253
Those are pooled criterion scores, a sixteen-percentage-point
gap within this reported comparison.

30
00:02:34.853 --> 00:02:39.293
Looking only at the overall number would hide which part of
understanding the task needs improvement.

31
00:02:39.860 --> 00:02:46.240
Return to the dinner plan. A polished answer can still overlook
a need that was present in the supplied context.

32
00:02:47.080 --> 00:02:52.940
xDailyBench makes those requirements explicit in evaluation,
even when they were implicit in the request.

33
00:02:53.760 --> 00:03:00.600
Its contribution is a way to ask whether an assistant connects
instructions, background, and a usable final result.

34
00:03:01.260 --> 00:03:06.840
The lesson is to read the whole task before deciding that the
visible words have been satisfied.