WEBVTT

1
00:00:00.260 --> 00:00:06.420
A deployed agent may receive a task instruction without a
ready-made skill library or reliable practice tests.

2
00:00:07.220 --> 00:00:12.480
OpenSkill asks how that agent can build both a useful procedure
and a way to assess it.

3
00:00:13.140 --> 00:00:18.400
Its adaptation lives in external skill artifacts, leaving the
underlying model weights unchanged.

4
00:00:19.040 --> 00:00:24.440
The pipeline uses public documentation, repositories, and other
open-world resources as learning material.

5
00:00:25.120 --> 00:00:31.080
The central design is to retrieve knowledge for the skill and
verifiable anchors for its tests as separate products.

6
00:00:31.780 --> 00:00:36.380
We will follow an original dataset-report example, then examine
what the measured outcomes support.

7
00:00:37.087 --> 00:00:42.187
Imagine a task that asks for a report from a public dataset,
with a specified output format.

8
00:00:42.827 --> 00:00:48.667
Knowledge retrieval can supply loading procedures, documented
library behavior, and useful domain conventions.

9
00:00:49.347 --> 00:00:52.327
A separate retrieval product supplies verification anchors:

10
00:00:52.907 --> 00:00:57.687
documented reference values, known dataset invariants, or
expected formats.

11
00:00:58.327 --> 00:01:04.647
In our illustration, a known row count and a documented table
schema give the checker concrete assertions to execute.

12
00:01:05.367 --> 00:01:11.387
Those checks assess properties of an output, rather than simply
asking the model whether its own answer looks good.

13
00:01:12.147 --> 00:01:18.267
The example contains no hidden benchmark answer. Its purpose is
to show how an external fact becomes a test.

14
00:01:18.913 --> 00:01:24.393
The retrieved knowledge becomes a structured plan describing the
skill architecture, procedures, and domain rules.

15
00:01:24.933 --> 00:01:30.173
The base agent drafts one to four skills, as specified by that
plan, and refines them together in one session.

16
00:01:30.713 --> 00:01:35.753
Because the skill is an external artifact, its procedure remains
readable and can be passed to another agent.

17
00:01:36.433 --> 00:01:41.093
In the report illustration, the skill would connect loading the
input, preserving relevant information,

18
00:01:41.273 --> 00:01:42.653
and writing the required output.

19
00:01:43.553 --> 00:01:49.213
The verifier is an isolated language-model session that produces
deterministic executable tests from the anchors.

20
00:01:49.733 --> 00:01:52.173
A separate session isolates the creator's conversation;

21
00:01:52.533 --> 00:01:55.413
it does not make proxy tests identical to the hidden target
tests.

22
00:01:55.900 --> 00:01:59.680
The current skill is executed and evaluated against the
self-built virtual suite.

23
00:02:00.260 --> 00:02:05.320
A failing assertion generates a diagnostic: what failed, why it
failed, and what should be revised.

24
00:02:06.240 --> 00:02:11.320
If the issue is a knowledge gap rather than an implementation
bug, the pipeline performs targeted retrieval.

25
00:02:12.160 --> 00:02:16.060
The default loop stops when the virtual suite passes or after
three refinement rounds,

26
00:02:16.180 --> 00:02:17.820
with additional budget and stall controls.

27
00:02:18.640 --> 00:02:21.740
The final deployed skill is the last refined version at
termination,

28
00:02:21.980 --> 00:02:24.500
rather than a best-of-many snapshot selected afterward.

29
00:02:25.300 --> 00:02:30.880
Virtual success is still a proxy. The later target evaluation
measures whether that practice produced useful behavior.

30
00:02:31.427 --> 00:02:36.147
The architecture hides ground-truth test definitions and
reference solutions during skill construction.

31
00:02:36.767 --> 00:02:41.327
Retrieval queries also filter benchmark names and identifiers
that could lead to hidden test material.

32
00:02:42.147 --> 00:02:47.507
The verifier receives the task environment and independent
anchors, with a separate session from the skill creator.

33
00:02:48.347 --> 00:02:52.527
Appendix F also describes a single ground-truth-derived
pass-or-fail bit

34
00:02:52.567 --> 00:02:54.747
when the verifier is asked to write deeper tests.

35
00:02:55.367 --> 00:03:00.747
It states that test contents, assertion details, and expected
values are not exposed in that invocation.

36
00:03:01.467 --> 00:03:03.547
The boundary therefore concerns hidden contents;

37
00:03:04.107 --> 00:03:08.407
the description does not justify saying every possible
ground-truth-derived signal is absent.

38
00:03:09.220 --> 00:03:13.840
The paper compares virtual-verifier decisions with ground-truth
outcomes on eighty-four tasks.

39
00:03:14.400 --> 00:03:20.220
Reported agreement is sixty point seven percent, with precision
fifty-six point nine and recall eighty point five.

40
00:03:20.800 --> 00:03:24.800
Here, the comparison distinguishes whether a task receives
positive ground-truth reward,

41
00:03:25.180 --> 00:03:27.360
rather than certifying every hidden requirement.

42
00:03:28.000 --> 00:03:32.180
In a separate fifteen-task sample, a language-model judge
matches virtual tests

43
00:03:32.260 --> 00:03:34.420
to the intended purposes of hidden tests.

44
00:03:34.960 --> 00:03:39.660
It finds one hundred twenty of one hundred thirty-five intents
covered, or eighty-eight point nine percent.

45
00:03:40.320 --> 00:03:42.780
Intent coverage and outcome agreement are different
measurements.

46
00:03:43.220 --> 00:03:46.160
Neither means a proxy pass is always a target-task pass.

47
00:03:46.820 --> 00:03:50.660
The main SkillsBench comparison covers eleven domains and two
target agents:

48
00:03:51.200 --> 00:03:52.780
Opus four point six with Claude Code,

49
00:03:53.220 --> 00:03:55.020
and GPT-five point two with Codex.

50
00:03:55.620 --> 00:04:00.480
For Opus, the reported overall pass rate rises from twenty-five
point five without skills

51
00:04:00.860 --> 00:04:03.120
to forty-three point six with OpenSkill.

52
00:04:03.700 --> 00:04:08.000
For GPT-five point two, it rises from twenty-five to forty-two
point one percent.

53
00:04:08.900 --> 00:04:14.980
These outcomes belong to the evaluated agent, framework, and
task setup, not a model-only causal comparison.

54
00:04:15.700 --> 00:04:19.920
SocialMaze and ScienceWorld provide two additional task types in
the paper's evaluation.

55
00:04:20.780 --> 00:04:23.640
The external skill artifact is frozen before target deployment,

56
00:04:24.080 --> 00:04:26.780
and target tests determine the resulting measured outcome.

57
00:04:27.267 --> 00:04:33.107
OpenSkill couples open-world learning with externally anchored
practice, while leaving model weights fixed.

58
00:04:33.667 --> 00:04:38.047
The report illustration shows the essential split: learn how to
produce the artifact,

59
00:04:38.227 --> 00:04:40.807
and independently specify properties worth checking.

60
00:04:41.367 --> 00:04:47.127
Diagnostics revise the procedure, and frozen skills carry that
learned procedure into later target execution.

61
00:04:47.787 --> 00:04:53.567
The paper also deploys Opus-generated skills to four other
models without retraining or regenerating those skills.

62
00:04:54.347 --> 00:04:59.667
Useful adaptation therefore needs inspectable skills, grounded
tests, and measured target outcomes,

63
00:05:00.067 --> 00:05:02.067
with the information boundary stated precisely.

64
00:05:02.627 --> 00:05:06.547
The practical question is whether the skill learned from
available resources helps

65
00:05:06.587 --> 00:05:08.867
when the real task finally has to be executed.