WEBVTT

1
00:00:00.320 --> 00:00:04.300
Imagine two agents using the same language model. One checks its
work,

2
00:00:04.500 --> 00:00:08.960
recovers from errors, and keeps useful context; the other stops
too early.

3
00:00:09.660 --> 00:00:14.740
Their surrounding software can change what the model
accomplishes. That software is the agent harness.

4
00:00:15.480 --> 00:00:21.360
HarnessDev asks whether a language model can build this
infrastructure, then improve the runnable system it created.

5
00:00:22.040 --> 00:00:27.240
The submitted object is persistent code, reused across tasks,
rather than one answer to one question.

6
00:00:27.840 --> 00:00:33.720
This changes the unit of evaluation: engineering the execution
system becomes the capability under test.

7
00:00:34.540 --> 00:00:39.840
We will follow a completion-check example, then separate
development feedback from the later held-out test.

8
00:00:40.380 --> 00:00:47.240
Creation begins with a deliberately weak seed that can read
configuration, expose tools, and write the required logs.

9
00:00:47.720 --> 00:00:55.560
It has no agent loop, task decomposition, verification, recovery
logic, or stopping rule. Unmodified, it scores zero.

10
00:00:56.240 --> 00:01:01.680
The creator receives a task-family specification, constraints,
and one to three development cases.

11
00:01:02.200 --> 00:01:06.740
It must connect the available primitives into a system that
actually attempts the downstream work.

12
00:01:07.320 --> 00:01:11.620
Hidden evaluation tasks and the human implementation remain
unavailable during development.

13
00:01:12.399 --> 00:01:18.500
A finished harness is then frozen before it is evaluated. The
benchmark measures what the added execution logic enables.

14
00:01:19.167 --> 00:01:25.867
The creator writes the harness. The executor is the model that
runs inside the frozen harness on downstream tasks.

15
00:01:26.627 --> 00:01:31.767
Keeping these roles separate lets the paper examine whether a
design depends on its original runtime model.

16
00:01:32.647 --> 00:01:36.887
Creation covers six creator models, four domains, and five
benchmarks,

17
00:01:37.087 --> 00:01:40.047
totaling two thousand two hundred seven unique instances.

18
00:01:40.947 --> 00:01:45.747
Generated harnesses show uneven performance: writing and
machine-learning experimentation

19
00:01:45.807 --> 00:01:49.647
are stronger than code and search relative to selected
references.

20
00:01:50.327 --> 00:01:55.467
Those reference comparisons are system-level comparisons, and
some reference scores come from external reports.

21
00:01:56.287 --> 00:02:01.527
Changing the executor can also change the rankings, exposing
compatibility as part of harness quality.

22
00:02:02.233 --> 00:02:05.753
One evolution trajectory provides a concrete diagnostic case.

23
00:02:06.453 --> 00:02:12.733
Opus finds that ninety-nine of one hundred runs report success,
but only forty-eight pass.

24
00:02:13.373 --> 00:02:18.233
It traces the gap to premature completion and adds a completion
check to the harness.

25
00:02:18.973 --> 00:02:23.913
Picture a gate before the final answer: has the requested
artifact been produced,

26
00:02:24.273 --> 00:02:26.573
and is there evidence that the task is complete?

27
00:02:27.173 --> 00:02:32.353
That question belongs to the execution logic surrounding the
model, rather than a change to its weights.

28
00:02:33.113 --> 00:02:38.793
The paper identifies this gate as a positive example with
execution-path and case-level evidence.

29
00:02:39.413 --> 00:02:45.413
The lesson is specific: useful feedback exposes a failure that
an executable change can address.

30
00:02:46.073 --> 00:02:51.633
Evolution starts from the creator's own frozen code harness and
repeatedly evaluates revised commits.

31
00:02:52.293 --> 00:02:57.093
Visible feedback uses one hundred SWE-Pro tasks and eighty-nine
Terminal-Bench tasks.

32
00:02:57.633 --> 00:03:00.333
An official candidate must finish both evaluation legs.

33
00:03:01.273 --> 00:03:07.053
After development ends, every official version is tested on six
hundred thirty disjoint SWE-Pro instances.

34
00:03:07.653 --> 00:03:12.093
These scores are never returned to the creator and cannot
influence its edits or final version choice.

35
00:03:12.833 --> 00:03:17.813
The feedback score averages two benchmark percentages; the
held-out score uses only SWE-Pro.

36
00:03:18.533 --> 00:03:23.753
A development gain and a held-out gain therefore answer
different questions. They must remain separate on the chart.

37
00:03:24.327 --> 00:03:28.167
Consider the GPT-five point five lineage executed by fixed
Gemini.

38
00:03:28.667 --> 00:03:32.787
Its feedback pair rises from fifty-six point six to fifty-nine
point one.

39
00:03:33.487 --> 00:03:40.627
On the held-out SWE-Pro set, the corresponding scores fall from
forty-two point two two to thirty-one point nine zero.

40
00:03:41.367 --> 00:03:46.707
The selected revision improved the visible pair while reducing
performance on that separate held-out set.

41
00:03:47.267 --> 00:03:52.087
This is one trajectory under one executor, not an average effect
attributed to that creator model.

42
00:03:52.827 --> 00:03:56.726
Across self-runtime declarations, all five improve held-out
performance;

43
00:03:57.167 --> 00:04:00.667
under fixed Gemini, only Opus improves and the other three
regress.

44
00:04:01.687 --> 00:04:06.467
Executor compatibility and generalization therefore belong in
the test of an evolved harness.

45
00:04:07.047 --> 00:04:11.687
The nine evolution lineages produce sixty-four adjacent official
version switches.

46
00:04:12.207 --> 00:04:17.587
Feedback and held-out scores move in the same direction on
thirty-four of those switches, about fifty-three percent.

47
00:04:18.167 --> 00:04:23.047
Only two of nine declared final versions are also the best
held-out versions within their lineages.

48
00:04:23.527 --> 00:04:29.367
Repeated runs of the same commit can vary, and many reported
gains remain inside the measured noise band.

49
00:04:29.907 --> 00:04:35.347
Selecting the highest visible score is therefore an unreliable
substitute for checking unseen tasks.

50
00:04:35.927 --> 00:04:41.287
The benchmark treats version selection as part of development,
rather than assuming every edit is an improvement.

51
00:04:41.740 --> 00:04:47.800
HarnessDev makes a model's surrounding execution system an
inspectable artifact that can persist across tasks.

52
00:04:48.440 --> 00:04:54.400
Creation asks whether the system can be built; Evolution asks
whether feedback can improve the existing artifact.

53
00:04:55.220 --> 00:04:58.220
The completion-gate example shows a constructive local repair.

54
00:04:58.720 --> 00:05:02.200
The held-out comparisons ask whether repairs travel beyond their
feedback set.

55
00:05:03.040 --> 00:05:07.860
Code must run, preserve useful behavior, and remain compatible
with the model that executes it.

56
00:05:08.680 --> 00:05:13.740
A convincing self-developed agent needs evidence about both its
software and its downstream performance.

57
00:05:14.340 --> 00:05:20.780
The important result is an engineering task we can measure:
build, freeze, run, inspect, and revise.