StructEval text-output mini-lab

A CPU-only, no-API-key example of what a structured-output score checks. Turn the order in fixtures/source.json into YAML while retaining its fields and values. Four handwritten teaching outputs show why parsing, field paths and values need separate checks. These are not model predictions or paper benchmark results.

This package runs two unmodified files from the official StructEval repository, pinned in provenance.json. It does not install model engines, generate outputs, render pages or reproduce the paper’s aggregate score.

Watch the two-minute walkthrough to see all four inputs and their verified outcomes.

Run

Download and extract the standalone package, then open a terminal in the extracted structeval-text directory:

python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python run.py --verify-fixtures

Windows: use .venv\Scripts\python.exe in place of .venv/bin/python. Tested locally with Python 3.9; Windows commands have not been executed. No network calls occur in run.py; dependency installation needs network access. Parsing the intentionally malformed JSON writes an expected error to stderr; the command still succeeds when all four results match.

Expected observations

The key-path score is computed by the pinned official evaluate_nonrenderable function. The wrong-value output receives a full path score because the expected paths exist. The last column is a deliberately separate application-level comparison written for this example; it is not a StructEval metric. It requires exact agreement with this one order, so it is not a general semantic evaluator either.

The score is not an overall reliability score or a judgment on StructEval’s other evaluation modes. See the paper and official full pipeline for generation, rendering, visual evaluation and benchmark settings.

Try your own outputs

Copy the fixtures directory to my-predictions, replace its four named output files with your own responses, and run:

.venv/bin/python run.py --predictions my-predictions

Preserve filenames and format types. This deliberately keeps one fixed input contract so you can compare outputs. A missing file is an input error, not a zero score. Keep --verify-fixtures for the unchanged reference fixtures only. verified-output.json is the retained local run, and expected.json is the hand-specified acceptance table.

This is a four-case learning exercise, not the proposed 20-example benchmark subset. A future actual subset should preserve upstream task IDs and report selection criteria; do not relabel these synthetic IDs as benchmark records.

Sources and attribution

• Paper and published TMLR version.
• Coauthor research guide and official repository.
• vendor/eval_utils.py and vendor/eval_nonrenderable.py are unchanged upstream files; their commit and SHA-256 hashes are recorded in provenance.json. Their Apache-2.0 license is in vendor/LICENSE.
• Wrapper, fixtures and explanation: Yuxuan Zhang, prepared with AI assistance, September 2026. Package contributions are Apache-2.0; see LICENSE. Contributions to the paper belong to its listed coauthors. This is a coauthor-maintained teaching package, not a new official benchmark release.

If you use the research in scholarly work, cite the paper version you used. No citation is required simply to try this exercise.