1
00:00:00,340 --> 00:00:03,420
Imagine a blue cup below a sign that reads hello.

2
00:00:04,180 --> 00:00:07,240
This illustrative example makes the evaluation problem visible.

3
00:00:07,940 --> 00:00:11,600
The instruction is simple: make the cup red, and keep the sign
unchanged.

4
00:00:12,400 --> 00:00:14,880
The request contains a change and a constraint.

5
00:00:15,820 --> 00:00:19,480
Candidate A changes the cup correctly, but drops a letter from
the sign.

6
00:00:20,260 --> 00:00:21,980
A quick glance at the color could miss that mistake.

7
00:00:23,000 --> 00:00:25,440
Candidate B changes the cup and preserves the word.

8
00:00:26,300 --> 00:00:30,580
The useful reward should reflect the whole instruction,
including what must stay the same.

9
00:00:31,580 --> 00:00:34,000
How can an evaluator learn what to inspect in an edit?

10
00:00:34,527 --> 00:00:38,827
RewardHarness approaches this question by evolving the evaluator
context.

11
00:00:39,307 --> 00:00:41,487
Instead of fitting the reward evaluator weights,

12
00:00:41,907 --> 00:00:46,247
it develops a library of written Skills and Tools from
preference demonstrations.

13
00:00:46,807 --> 00:00:49,427
The evaluator is a frozen vision language model.

14
00:00:49,727 --> 00:00:53,307
The default is Qwen two point five VL, with seven billion
parameters.

15
00:00:53,807 --> 00:00:56,267
What changes is the guidance placed in front of it:

16
00:00:56,707 --> 00:01:01,387
criteria to apply, examples to follow, and targeted procedures
to perform.

17
00:01:01,987 --> 00:01:04,187
Think of a reviewer receiving a better checklist.

18
00:01:04,547 --> 00:01:07,147
The reviewer stays the same; the guidance improves.

19
00:01:07,727 --> 00:01:10,467
A Skill is a structured Markdown guideline.

20
00:01:10,887 --> 00:01:13,447
It names the evaluation task, describes it,

21
00:01:13,967 --> 00:01:17,587
and breaks quality into a scoring rubric supported by examples.

22
00:01:18,427 --> 00:01:23,627
The paper gives a revealing example: a rubric distinguishes
visible image artifacts

23
00:01:23,687 --> 00:01:27,687
from conceptual unrealism that the instruction actually
requests.

24
00:01:28,867 --> 00:01:30,387
A Tool supplies the procedure.

25
00:01:30,927 --> 00:01:34,967
It states its purpose, expected inputs and outputs, when to use
it,

26
00:01:35,287 --> 00:01:37,867
and the steps for a focused visual analysis.

27
00:01:38,647 --> 00:01:43,947
For text inspection, the vision language model reads a protocol
to extract and compare lettering.

28
00:01:44,527 --> 00:01:47,507
The Tool is that written protocol, followed by the model.

29
00:01:48,407 --> 00:01:53,047
Our cup example needs a preservation criterion and a careful
inspection of the sign.

30
00:01:53,653 --> 00:01:57,893
At inference time, the orchestrator examines the editing
instruction and source image,

31
00:01:58,373 --> 00:02:01,033
then selects relevant Skills and Tools from the library.

32
00:02:01,913 --> 00:02:04,933
Those documents become the context for the frozen evaluator.

33
00:02:05,793 --> 00:02:10,793
The same evaluator can now attend to the criteria and procedures
selected for this particular edit.

34
00:02:11,573 --> 00:02:17,073
First, it applies the Skill rubrics to each candidate and
records assessments for the individual criteria.

35
00:02:17,913 --> 00:02:20,213
Next, when a Tool meets its invocation conditions,

36
00:02:20,613 --> 00:02:23,113
it follows that protocol for a targeted analysis,

37
00:02:23,653 --> 00:02:26,013
such as checking text or spatial relationships.

38
00:02:26,953 --> 00:02:31,273
Finally, it combines assessments and analysis outputs into a
preference ranking,

39
00:02:31,693 --> 00:02:33,553
supported by the explicit reasoning chain.

40
00:02:34,080 --> 00:02:38,600
The library is learned from a calibration set of one hundred
human preference demonstrations.

41
00:02:39,180 --> 00:02:42,360
The orchestrator divides it into about seventy training examples

42
00:02:42,420 --> 00:02:44,460
and thirty held out validation examples.

43
00:02:45,360 --> 00:02:48,800
On the training portion, it compares predicted rankings with the
human preferences

44
00:02:49,140 --> 00:02:51,920
and examines the reasoning traces behind successes and errors.

45
00:02:52,760 --> 00:02:54,680
A missing criterion can motivate a new Skill.

46
00:02:55,460 --> 00:02:57,240
A faulty assessment can motivate a revision.

47
00:02:58,000 --> 00:03:00,800
A perceptual mistake can motivate a better Tool procedure.

48
00:03:01,740 --> 00:03:03,800
The proposed edit goes into a candidate library.

49
00:03:04,500 --> 00:03:09,500
It becomes the committed version only if validation accuracy
strictly improves over the current library.

50
00:03:10,300 --> 00:03:13,500
Otherwise, the candidate is discarded and the previous snapshot
is restored.

51
00:03:14,620 --> 00:03:17,360
After the evolution budget, the final library is frozen.

52
00:03:18,200 --> 00:03:21,580
Learning means retaining useful written context through the
validation gate.

53
00:03:22,227 --> 00:03:26,027
Table one gives a direct comparison under the same Qwen
backbone.

54
00:03:26,447 --> 00:03:30,687
The raw seven billion parameter model has average accuracy of
thirty point three percent.

55
00:03:31,327 --> 00:03:35,047
EditReward with that Qwen backbone reaches forty two point zero
percent.

56
00:03:35,827 --> 00:03:40,147
RewardHarness with the frozen Qwen evaluator reaches forty five
point seven percent.

57
00:03:40,987 --> 00:03:43,247
These averages combine four evaluation columns:

58
00:03:43,727 --> 00:03:46,947
three EditReward Bench candidate settings and GenAI Bench.

59
00:03:47,487 --> 00:03:51,787
They are arithmetic means of the columns, not pooled accuracy
across all comparisons.

60
00:03:52,607 --> 00:03:56,727
A separately evaluated version using Gemini two point zero Flash
as the sub agent

61
00:03:56,767 --> 00:03:58,707
reaches forty seven point four percent.

62
00:03:59,287 --> 00:04:01,467
That is the evaluator variant named in the paper.

63
00:04:02,107 --> 00:04:08,527
Calibration examples come from the training split; EditReward
Bench is disjoint and reserved for final evaluation.

64
00:04:09,167 --> 00:04:12,047
The comparison tests evolved context on separate data.

65
00:04:12,587 --> 00:04:15,327
The paper then runs a separate downstream experiment.

66
00:04:15,847 --> 00:04:19,127
RewardHarness supplies the reward signal for Flow GRPO training

67
00:04:19,187 --> 00:04:22,227
of the FLUX two klein base editor with four billion parameters.

68
00:04:22,907 --> 00:04:27,827
Here, the image editor is trained, in a separate stage from
evolving documents for the frozen evaluator.

69
00:04:28,687 --> 00:04:32,927
On ImgEdit Bench, the base editor receives an overall score of
three point three two.

70
00:04:33,567 --> 00:04:38,187
Under the same Flow GRPO setup, EditReward training reaches
three point four five,

71
00:04:38,587 --> 00:04:41,027
and RewardHarness training reaches three point five two.

72
00:04:41,687 --> 00:04:43,747
These are overall benchmark scores in points.

73
00:04:44,447 --> 00:04:47,387
The reported comparison connects the learned evaluation context

74
00:04:47,747 --> 00:04:51,107
to a larger overall editing improvement in this controlled
training setup.

75
00:04:51,827 --> 00:04:53,387
Return to the cup and the sign.

76
00:04:54,047 --> 00:05:00,127
A preference judgment becomes more useful when it checks the
requested change and the details that should survive it.

77
00:05:01,047 --> 00:05:04,607
RewardHarness turns those evaluation habits into written
artifacts:

78
00:05:05,267 --> 00:05:09,527
Skills express the criteria, Tools describe the inspection
procedures,

79
00:05:10,127 --> 00:05:12,587
and the orchestrator chooses the relevant context.

80
00:05:13,307 --> 00:05:16,047
The validation loop decides which changes deserve to stay.

81
00:05:16,727 --> 00:05:21,167
Its output is a frozen library, ready to guide the evaluator on
future edits.

82
00:05:22,247 --> 00:05:27,207
The earned takeaway: improving the context an evaluator reads
can improve its judgment

83
00:05:27,567 --> 00:05:29,047
while keeping the backbone fixed.

84
00:05:29,767 --> 00:05:31,667
Better criteria and procedures matter.