Creating and evolving model-external agent harnesses

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Wu, Yuhao, Zhang, Jingyuan, Shi, Jiajun, Lei, Xinping, Gu, Qingshui, Zhang, Yuxuan, Wang, Zexuan, He, Chen, Huang, Chen, Song, Maojia, Zeng, Zhiyuan, Wang, Shaowen, Liu, Jinkai, Shi, Yunfeng, Liu, Jiaheng, Yan, Shen, Huang, Wenhao, Zhang, Ge, Zhang, Wenxuan

Paper / source record · 2026-09-01

arXiv Project

Research question and approach

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored.

When this work is relevant

Cite this benchmark when evaluating creation or evolution of agent execution infrastructure, keeping harness changes distinct from model-weight updates.

Author-written abstract

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.

Abstract source: https://arxiv.org/abs/2609.01437. Checked 2026-09-14. Bibliographic metadata uses the linked paper record or author-maintained catalog. Results, limitations and experimental settings remain defined by the original source.

Citation

BibTeX · CITATION.cff

BibTeX is preserved from the author-maintained citation repository. The CFF uses the source metadata shown on this page. Version titles or author lists can differ; choose the version you used.

@article{wu2026harnessdev,
  title={HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?},
  author={Wu, Yuhao and Zhang, Jingyuan and Shi, Jiajun and Lei, Xinping and Gu, Qingshui and Zhang, Yuxuan and Wang, Zexuan and He, Chen and Huang, Chen and Song, Maojia and others},
  journal={arXiv preprint arXiv:2609.01437},
  year={2026}
}