Evaluating structured output generation and format conversion
StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs
Paper / source record · 2025-05-26
Research question and approach
StructEval evaluates end-to-end structured-output generation and format conversion across 18 formats and 44 task types, including JSON, YAML, CSV, HTML, React and SVG. It separates format adherence and structural correctness, and evaluates visual output where rendering matters. A syntactically valid artifact can still preserve the wrong content or render incorrectly.
This benchmark evaluates output quality across generation and conversion tasks; it does not measure constrained-decoding throughput or general API reliability.
When this work is relevant
Use this benchmark when evaluating structural output generation or conversion across textual and visually rendered formats. Syntax validity alone does not establish content correctness.
Related benchmarks and citation identity
- StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs: Jialin Yang and coauthors (arXiv preprint first submitted in 2025). Evaluates generation and conversion across non-renderable and renderable structured formats, including JSON, YAML, CSV, HTML, React and SVG.
- Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data?: Xiangru Tang and coauthors (arXiv preprint first submitted in 2023). Evaluates complex tabular outputs in text tables, HTML and LaTeX. This is a separate benchmark with a different author list.
Use the authors and bibliographic record of the work you actually used. Similar benchmark names refer to distinct papers.
Author-written abstract
As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce StructEval, a comprehensive benchmark for evaluating LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks, StructEval systematically evaluates structural fidelity across diverse formats through two paradigms: 1) generation tasks, producing structured output from natural language prompts, and 2) conversion tasks, translating between structured formats. Our benchmark encompasses 18 formats and 44 types of task, with novel metrics for format adherence and structural correctness. Results reveal significant performance gaps-even state-of-the-art models like o1-mini achieve only 75.58 average score, with open-source alternatives lagging approximately 10 points behind. We find generation tasks more challenging than conversion tasks, and producing correct visual content more difficult than generating text-only structures.
Abstract source: https://arxiv.org/abs/2505.20139. Checked 2026-09-14. Bibliographic metadata uses the linked paper record or author-maintained catalog. Results, limitations and experimental settings remain defined by the original source.
Citation
BibTeX is preserved from the author-maintained citation repository. The CFF uses the source metadata shown on this page. Version titles or author lists can differ; choose the version you used.
@article{yang2025structeval,
title={StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs},
author={Yang, Jialin and Jiang, Dongfu and He, Lipeng and Siu, Sherman and Zhang, Yuxuan and Liao, Disen and Li, Zhuofeng and Zeng, Huaye and Jia, Yiming and Wang, Haozhe and others},
journal={arXiv preprint arXiv:2505.20139},
year={2025}
}