Valid YAML, Wrong Order

A two-minute walkthrough of four outputs: correct YAML, a missing field, a changed value and malformed JSON.

Run the mini-lab · Download MP4 · SRT subtitles

Scope: This narrated walkthrough displays handwritten fixtures and locally verified evaluator outputs. It is not a screen recording, a model experiment or a reproduction of aggregate benchmark results. The extra exact-value comparison belongs to this teaching application; it is not a StructEval metric.

Attribution: Yuxuan Zhang coauthored StructEval. Script and visuals were prepared with AI assistance; narration is synthetic. Research contributions belong to all authors listed in the original paper. The mini-lab uses unchanged official evaluator files pinned to commit 13033f28125a03d9348e026d7530957eb7087593.

Try the same checks

Download and extract the standalone package, then run these commands inside its directory:

python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python run.py --verify-fixtures

The malformed JSON case deliberately logs a parsing error. Verification succeeds when all four observed outcomes match the fixtures. See the mini-lab README for trying your own predictions and platform notes.

Original paper · Published TMLR version · Official repository · Research guide and authors · Companion bilingual article

Full transcript

00:00 — Valid YAML. Wrong order.

A format conversion can produce valid YAML and still change your order. This walkthrough uses four handwritten outputs from our public StructEval mini lab. We run the pinned official key path evaluator, then a separate exact value check. These are teaching examples, not model predictions or paper benchmark results.

00:19 — Start with the input contract

The source order contains an identifier, one product name, and a quantity of two. This YAML retains all three required paths and their values. It parses, receives a key path score of one, and passes our additional exact value comparison. Start by making the expected input contract explicit.

00:38 — Parsing does not check completeness

Now remove the quantity field. The YAML remains syntactically valid, but one of the three expected paths is missing. The official path score falls to two thirds. The additional value comparison fails too. A parser answers whether the format can be read. It does not establish that every required field survived.

00:57 — A full path score can miss a changed value

Next, change the quantity from two to two hundred. All expected paths still exist, so this key path score remains one. Our separate exact value check catches the change. This is a limit of the particular check demonstrated here. It is not an overall assessment of the full StructEval evaluation pipeline.

01:15 — Reject broken syntax before checking content

The fourth output is JSON with trailing commas. Parsing fails, and the path score is zero. The example deliberately includes this failure, so an error message is expected. The fixture verification command succeeds when all four observed results match their expected outcomes. A failing test case is different from a broken test runner.

01:36 — Run the checks on your own outputs

Download and extract the package from the linked mini lab. Create an environment, install its dependencies, and run the fixture verification command shown here. To try your own responses, copy the fixtures and use the predictions option. The value comparison is application code, not a StructEval metric. This coauthor guide uses synthetic narration and AI assistance.