StructEval: Benchmarking LLMs’ Capabilities to Generate Structural Outputs

Paper: StructEval: Benchmarking LLMs’ Capabilities to Generate Structural Outputs (v3) https://arxiv.org/html/2505.20139v3

This explainer follows the mechanism before interpreting its reported results.

Suppose a request asks for two blue objects with separate color and count fields. A JSON object can have valid punctuation and a count of two while setting the color to red. Likewise, an SVG can render successfully and display the wrong circle. Our original examples demonstrate a contract mismatch; they are not measured model outputs.

StructEval evaluates that distinction between structure and meaning. The benchmark contains 2,035 examples across 18 formats and 44 task types. Four categories separate text and rendered outputs, and generation and conversion. Generation constructs the requested structure; conversion must preserve the relevant content and relationships while changing formats.

The evaluation therefore depends on what an output is supposed to do. For text structures, a binary parsing check measures syntax. Keyword-based checks inspect required fields, values and nested paths. The composite weights are 20% syntax and 80% keyword checks, so parsing alone cannot earn a complete score.

Rendered structures add another question: what does the code actually display? Their evaluation combines execution, keyword checks and reference visual question-and-answer checks by a vision-language judge. The respective weights are 20%, 10% and 70%. A successful render is only one part of the contract.

After synthesis, the benchmark’s tasks and metrics receive two expert review passes. The reported model comparison uses a uniform zero-shot, greedy setup, with GPT-4.1 mini as the visual judge. These details describe the evaluation configuration rather than an unqualified ranking of every model.

Table 6 reports four-category composite scores of 76.02 points for GPT-4o and 67.04 points for Qwen3-4B. They are not percentages of fully correct outputs. The metric includes differently weighted checks across tasks.

The paper gives a concrete evaluation habit: parse the structure, verify the requested relationships, and inspect the displayed meaning when the format is visual. Syntax is a necessary observation, but the output still has to satisfy the request.

Paper: StructEval: Benchmarking LLMs’ Capabilities to Generate Structural Outputs (v3).