WEBVTT 1 00:00:00.280 --> 00:00:02.380 Become better at mathematical reasoning. 2 00:00:02.780 --> 00:00:08.560 That sounds like a goal, but it does not say which examples to study or what progress should look like. 3 00:00:09.260 --> 00:00:15.140 Aspire studies this missing step: can an agent decide what to learn, as well as how to update itself? 4 00:00:15.800 --> 00:00:21.120 Here, vague means a broad capability direction, not a careless or ambiguous instruction. 5 00:00:21.940 --> 00:00:27.500 The agent must turn that direction into data choices, an update objective, and a way to check its own work. 6 00:00:28.180 --> 00:00:34.080 The final evaluation items stay hidden. They test whether the agent's chosen proxy serves the intended capability. 7 00:00:34.860 --> 00:00:38.960 We will follow that decision chain, then compare apparent progress with the starting model. 8 00:00:39.600 --> 00:00:45.480 In an illustrative mathematics campaign, the agent might diagnose a weakness in solving multi-step algebra problems. 9 00:00:46.000 --> 00:00:50.560 It then chooses practice data, a training objective, and its own validation questions. 10 00:00:51.240 --> 00:00:54.860 But suppose its local checks mainly reward a familiar answer format. 11 00:00:55.340 --> 00:00:57.600 Getting better at that check may miss the original goal. 12 00:00:58.240 --> 00:01:02.720 Aspire makes this distinction explicit: progress on an agent-built proxy 13 00:01:02.860 --> 00:01:06.000 must be assessed against a goal-aligned external criterion. 14 00:01:06.680 --> 00:01:12.320 Its managed tool interface supports data selection, training, local validation, branching, and stopping. 15 00:01:12.880 --> 00:01:17.640 Backend job mechanics are handled by the controller, leaving the strategic learning decisions to the agent. 16 00:01:18.300 --> 00:01:23.440 The benchmark supports two evolution surfaces: model weights and the supporting agent harness. 17 00:01:24.200 --> 00:01:29.140 Weight evolution selects data and an update recipe, then produces a trained descendant checkpoint. 18 00:01:29.840 --> 00:01:35.560 Harness evolution edits instructions, tool policy, workflow, memory, or validation logic 19 00:01:35.600 --> 00:01:37.000 while runtime weights stay fixed. 20 00:01:37.740 --> 00:01:41.320 The policy directing the search is distinguished from the object being changed. 21 00:01:42.140 --> 00:01:45.420 All reported weight-search runs keep their initial decision model fixed; 22 00:01:45.780 --> 00:01:48.800 they do not demonstrate recursive replacement of that decision model. 23 00:01:49.480 --> 00:01:53.100 This lets the study ask whether a bounded search creates a better candidate 24 00:01:53.160 --> 00:01:55.800 without confusing execution with capability growth. 25 00:01:56.380 --> 00:02:02.580 The hidden Aspire evaluation contains five hundred twenty expert-authored items across six capability goals. 26 00:02:03.060 --> 00:02:09.220 These cover science, humanities, medicine, mathematics, integrated reliability, and academic writing, 27 00:02:09.700 --> 00:02:11.080 with each goal scored separately. 28 00:02:11.760 --> 00:02:17.460 The agent cannot inspect evaluation items, reference answers, rubrics, or per-item judge feedback. 29 00:02:18.140 --> 00:02:22.960 In adaptive feedback, bounded queries return aggregate scores that can guide later updates. 30 00:02:23.640 --> 00:02:30.180 That setting uses the same goal evaluation slice repeatedly. It is not an untouched confirmation test after selection. 31 00:02:30.840 --> 00:02:35.960 A separate final-only protocol withholds intermediate scores. Its outcomes are reported separately. 32 00:02:36.620 --> 00:02:41.700 In the adaptive-feedback experiment, thirty configuration-goal cells receive one run each. 33 00:02:42.380 --> 00:02:44.200 Twenty-eight produce an evaluated checkpoint, 34 00:02:44.580 --> 00:02:48.100 and twenty-one produce a checkpoint meeting the predeclared eligibility rules. 35 00:02:48.840 --> 00:02:53.640 Only one cell produces an eligible above-base checkpoint that remains after the rollback rule. 36 00:02:54.440 --> 00:02:58.680 Eligibility includes provenance, exploration, completion, and a terminal record. 37 00:02:59.140 --> 00:03:01.140 It does not itself mean a capability gain. 38 00:03:01.920 --> 00:03:05.700 The retained result was selected using repeated aggregate feedback on the same slice, 39 00:03:06.060 --> 00:03:07.740 without a separate confirmation slice. 40 00:03:08.440 --> 00:03:13.800 The measured finding is sparse retained progress under this protocol, alongside many completed training loops. 41 00:03:14.593 --> 00:03:19.413 One Qwen three point five four-B mathematics trajectory makes the distinction concrete. 42 00:03:19.913 --> 00:03:25.973 Its updated checkpoints score zero point seven nine, then one point seven five, then two point three eight. 43 00:03:26.593 --> 00:03:32.273 Those scores rise, but the starting model scored seventeen point eight six on the same goal slice. 44 00:03:32.973 --> 00:03:38.333 The later checkpoint is better than an earlier damaged checkpoint, while remaining well below the base. 45 00:03:38.973 --> 00:03:43.773 Aspire therefore distinguishes the best evaluated candidate, an eligible selected candidate, 46 00:03:44.193 --> 00:03:46.033 and the state retained after rollback. 47 00:03:46.613 --> 00:03:50.613 A positive slope within the lineage cannot replace an above-base comparison. 48 00:03:51.140 --> 00:03:55.440 The harness study holds Qwen three point five four-B runtime weights fixed 49 00:03:55.500 --> 00:03:57.840 and evaluates academic and scientific writing. 50 00:03:58.540 --> 00:04:00.260 Each creator attempts one successor. 51 00:04:00.980 --> 00:04:05.140 Valid harnesses are frozen, then executed three times on twenty hidden writing items. 52 00:04:06.000 --> 00:04:10.460 The reference Qwen-Agent scores twenty-eight point six four on task-macro evaluation. 53 00:04:11.000 --> 00:04:14.160 The closest valid successor reaches twenty-seven point two two. 54 00:04:15.080 --> 00:04:19.200 All three valid successors are numerically below that reference under the fixed runtime. 55 00:04:20.000 --> 00:04:25.020 The repeated executions measure runtime variation, not three independent creation trajectories. 56 00:04:25.860 --> 00:04:31.720 This experiment tests one-step harness generation, rather than an indefinitely repeating self-improvement process. 57 00:04:32.307 --> 00:04:35.986 Aspire brings target formation into the self-evolution problem: 58 00:04:36.607 --> 00:04:40.447 deciding what progress means comes before searching for an update. 59 00:04:41.187 --> 00:04:47.307 An agent can collect data, launch training, and produce a checkpoint without improving the intended capability. 60 00:04:48.026 --> 00:04:53.207 It can also construct a working harness whose hidden-evaluation score is lower than the starting reference. 61 00:04:54.047 --> 00:05:00.027 The useful chain is goal, proxy, candidate, external measurement, and retained outcome. 62 00:05:00.947 --> 00:05:06.747 Each link has to stay visible, and improvement must be judged against the base, not only the previous checkpoint. 63 00:05:07.707 --> 00:05:12.307 Running the learning loop is a start. Establishing that it learned the right thing is the test.