All the fields are there. The value is still wrong.

An order asks for two notebooks. A YAML response includes the order ID, item name and quantity, but changes the quantity to 200. It parses, and every required field exists. Which check should fail?

I coauthored StructEval. This note was prepared with AI assistance and describes four handwritten teaching outputs, not model predictions. The results below come from a small, locally executed exercise using pinned official evaluator code. They are not the paper’s benchmark results.

Name the property before interpreting the score

A parser checks whether the output is acceptable to that parser. A path check asks whether required locations exist. Neither alone establishes that values match the input. In this exercise the required paths are order.id, order.items[0].name and order.items[0].quantity.

The wrong-value case is not a bug in a metric defined to check path existence. It shows why the metric’s scope matters. The exact-value column is an additional application check written for this order, not a StructEval metric. Exact agreement can itself be too strict when multiple outputs are valid, so define the accepted alternatives for your application.

Make this useful in an evaluation

1. Write down the input contract. Identify required fields, allowed values, types and relationships.
2. Create deliberate failures before evaluating a model. A missing key and an incorrect value should exercise different checks.
3. Keep the checks separate in the report. Explain what a pass establishes and what it leaves unresolved.
4. Save the input, output, evaluator version and failure details together. Avoid turning a successful parse into an overall reliability claim.

For visual output, a screenshot introduces different questions; for an interactive page, behavior requires further tests. This small text exercise does not cover either mode.

Run it and replace an output

Download the CPU mini-lab. It includes inputs, expected results, dependency versions, unmodified official files and their SHA-256 hashes, licenses, and the retained local output. No model API or GPU is needed. The published ZIP was downloaded without authentication, extracted, and checked from a different directory; all four expected cases passed.

Try changing the quantity in a supplied file, then inspect the two columns independently. This is a learning exercise, not an independent reproduction of the full benchmark or evidence of external adoption.

StructEval paper · Published version · Pinned evaluator · Research guide and citation files