1
00:00:00,320 --> 00:00:04,300
Imagine two agents using the same language model. One checks its
work,

2
00:00:04,500 --> 00:00:08,960
recovers from errors, and keeps useful context; the other stops
too early.

3
00:00:09,660 --> 00:00:14,740
Their surrounding software can change what the model
accomplishes. That software is the agent harness.

4
00:00:15,480 --> 00:00:21,360
HarnessDev asks whether a language model can build this
infrastructure, then improve the runnable system it created.

5
00:00:22,040 --> 00:00:27,240
The submitted object is persistent code, reused across tasks,
rather than one answer to one question.

6
00:00:27,840 --> 00:00:33,720
This changes the unit of evaluation: engineering the execution
system becomes the capability under test.

7
00:00:34,540 --> 00:00:39,840
We will follow a completion-check example, then separate
development feedback from the later held-out test.

8
00:00:40,380 --> 00:00:47,240
Creation begins with a deliberately weak seed that can read
configuration, expose tools, and write the required logs.

9
00:00:47,720 --> 00:00:55,560
It has no agent loop, task decomposition, verification, recovery
logic, or stopping rule. Unmodified, it scores zero.

10
00:00:56,240 --> 00:01:01,680
The creator receives a task-family specification, constraints,
and one to three development cases.

11
00:01:02,200 --> 00:01:06,740
It must connect the available primitives into a system that
actually attempts the downstream work.

12
00:01:07,320 --> 00:01:11,620
Hidden evaluation tasks and the human implementation remain
unavailable during development.

13
00:01:12,399 --> 00:01:18,500
A finished harness is then frozen before it is evaluated. The
benchmark measures what the added execution logic enables.

14
00:01:19,167 --> 00:01:25,867
The creator writes the harness. The executor is the model that
runs inside the frozen harness on downstream tasks.

15
00:01:26,627 --> 00:01:31,767
Keeping these roles separate lets the paper examine whether a
design depends on its original runtime model.

16
00:01:32,647 --> 00:01:36,887
Creation covers six creator models, four domains, and five
benchmarks,

17
00:01:37,087 --> 00:01:40,047
totaling two thousand two hundred seven unique instances.

18
00:01:40,947 --> 00:01:45,747
Generated harnesses show uneven performance: writing and
machine-learning experimentation

19
00:01:45,807 --> 00:01:49,647
are stronger than code and search relative to selected
references.

20
00:01:50,327 --> 00:01:55,467
Those reference comparisons are system-level comparisons, and
some reference scores come from external reports.

21
00:01:56,287 --> 00:02:01,527
Changing the executor can also change the rankings, exposing
compatibility as part of harness quality.

22
00:02:02,233 --> 00:02:05,753
One evolution trajectory provides a concrete diagnostic case.

23
00:02:06,453 --> 00:02:12,733
Opus finds that ninety-nine of one hundred runs report success,
but only forty-eight pass.

24
00:02:13,373 --> 00:02:18,233
It traces the gap to premature completion and adds a completion
check to the harness.

25
00:02:18,973 --> 00:02:23,913
Picture a gate before the final answer: has the requested
artifact been produced,

26
00:02:24,273 --> 00:02:26,573
and is there evidence that the task is complete?

27
00:02:27,173 --> 00:02:32,353
That question belongs to the execution logic surrounding the
model, rather than a change to its weights.

28
00:02:33,113 --> 00:02:38,793
The paper identifies this gate as a positive example with
execution-path and case-level evidence.

29
00:02:39,413 --> 00:02:45,413
The lesson is specific: useful feedback exposes a failure that
an executable change can address.

30
00:02:46,073 --> 00:02:51,633
Evolution starts from the creator's own frozen code harness and
repeatedly evaluates revised commits.

31
00:02:52,293 --> 00:02:57,093
Visible feedback uses one hundred SWE-Pro tasks and eighty-nine
Terminal-Bench tasks.

32
00:02:57,633 --> 00:03:00,333
An official candidate must finish both evaluation legs.

33
00:03:01,273 --> 00:03:07,053
After development ends, every official version is tested on six
hundred thirty disjoint SWE-Pro instances.

34
00:03:07,653 --> 00:03:12,093
These scores are never returned to the creator and cannot
influence its edits or final version choice.

35
00:03:12,833 --> 00:03:17,813
The feedback score averages two benchmark percentages; the
held-out score uses only SWE-Pro.

36
00:03:18,533 --> 00:03:23,753
A development gain and a held-out gain therefore answer
different questions. They must remain separate on the chart.

37
00:03:24,327 --> 00:03:28,167
Consider the GPT-five point five lineage executed by fixed
Gemini.

38
00:03:28,667 --> 00:03:32,787
Its feedback pair rises from fifty-six point six to fifty-nine
point one.

39
00:03:33,487 --> 00:03:40,627
On the held-out SWE-Pro set, the corresponding scores fall from
forty-two point two two to thirty-one point nine zero.

40
00:03:41,367 --> 00:03:46,707
The selected revision improved the visible pair while reducing
performance on that separate held-out set.

41
00:03:47,267 --> 00:03:52,087
This is one trajectory under one executor, not an average effect
attributed to that creator model.

42
00:03:52,827 --> 00:03:56,726
Across self-runtime declarations, all five improve held-out
performance;

43
00:03:57,167 --> 00:04:00,667
under fixed Gemini, only Opus improves and the other three
regress.

44
00:04:01,687 --> 00:04:06,467
Executor compatibility and generalization therefore belong in
the test of an evolved harness.

45
00:04:07,047 --> 00:04:11,687
The nine evolution lineages produce sixty-four adjacent official
version switches.

46
00:04:12,207 --> 00:04:17,587
Feedback and held-out scores move in the same direction on
thirty-four of those switches, about fifty-three percent.

47
00:04:18,167 --> 00:04:23,047
Only two of nine declared final versions are also the best
held-out versions within their lineages.

48
00:04:23,527 --> 00:04:29,367
Repeated runs of the same commit can vary, and many reported
gains remain inside the measured noise band.

49
00:04:29,907 --> 00:04:35,347
Selecting the highest visible score is therefore an unreliable
substitute for checking unseen tasks.

50
00:04:35,927 --> 00:04:41,287
The benchmark treats version selection as part of development,
rather than assuming every edit is an improvement.

51
00:04:41,740 --> 00:04:47,800
HarnessDev makes a model's surrounding execution system an
inspectable artifact that can persist across tasks.

52
00:04:48,440 --> 00:04:54,400
Creation asks whether the system can be built; Evolution asks
whether feedback can improve the existing artifact.

53
00:04:55,220 --> 00:04:58,220
The completion-gate example shows a constructive local repair.

54
00:04:58,720 --> 00:05:02,200
The held-out comparisons ask whether repairs travel beyond their
feedback set.

55
00:05:03,040 --> 00:05:07,860
Code must run, preserve useful behavior, and remain compatible
with the model that executes it.

56
00:05:08,680 --> 00:05:13,740
A convincing self-developed agent needs evidence about both its
software and its downstream performance.

57
00:05:14,340 --> 00:05:20,780
The important result is an engineering task we can measure:
build, freeze, run, inspect, and revise.