S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?

Paper: S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement? (v1) https://arxiv.org/html/2608.31100v1

S3Gym: what should experience preserve?

Experience is useful when the next action changes for a reason that still applies. S3Gym makes gathering evidence, judging it and transferring it separate evaluation stages.

The original Minesweeper illustration starts with a clue of one and a genuinely known adjacent mine. The remaining unknown neighbors are safe because that local constraint is already satisfied. Remembering one clicked coordinate loses the reason it was safe. A rule can be applied to a new board only after inspecting its current state. This is schematic teaching material, not a reported trajectory.

The protocol separates more permissive exploration from stricter evaluation, using disjoint random seeds. Evaluation trajectories never enter reused experience. In the main context experiment, thirty exploration episodes are evaluated every three episodes using three strict episodes per checkpoint. Agent-generated scores influence reuse, while verifier-computed rewards remain benchmark-side during exploration.

History in-context learning retains detailed observations and actions. Summary Memory compresses experience into rules, mistakes and future directions. Parameter Training converts selected or corrected experience into supervised updates. Each has a different information bottleneck. Accurate local self-judgment alone is not a reliable substitute for measuring next-checkpoint improvement.

Across seven paired normalized above-baseline comparisons, Tetris has five summary wins, one history win and one tie. Minesweeper has three summary wins and four history wins. A separate Qwen3-8B parameter study rises from zero to a peak Trust Evolution score of thirty, while Plants-versus-Zombies drops from 23 to six at every updated checkpoint. Those games use different scales. The lesson is to check what experience preserves and whether it transfers, rather than assuming more memory or another update is improvement.

Watch the companion English animation with external English, Simplified Chinese or bilingual captions.

Paper, exact arXiv v1: https://arxiv.org/html/2608.31100v1

#AIAgents #AgentLearning #AIResearch