WEBVTT

1
00:00:00.340 --> 00:00:07.040
Suppose the request asks for structured data describing two blue
objects, with separate fields for color and count.

2
00:00:07.780 --> 00:00:13.480
Our first JSON object has valid punctuation and a count of two,
but its color value is red.

3
00:00:14.100 --> 00:00:20.640
A parser can read it successfully, so syntactic validity alone
would not reveal the mismatch with the requested content.

4
00:00:21.540 --> 00:00:26.680
Now compare a corrected object that keeps the same structure but
changes the color value to blue.

5
00:00:27.400 --> 00:00:33.520
Both objects are original teaching examples, not benchmark
submissions or measured outputs from an evaluated model.

6
00:00:34.340 --> 00:00:40.120
Readable format and required structure are different
achievements, motivating the distinctions measured by
StructEval.

7
00:00:40.613 --> 00:00:46.573
For a visual example, ask for one blue circle described by SVG
code that can be rendered.

8
00:00:47.253 --> 00:00:52.913
The candidate code is well formed and draws a circle, but the
fill color is red rather than blue.

9
00:00:53.773 --> 00:00:59.073
Successful rendering proves execution, while the displayed
circle still violates the requested color.

10
00:00:59.933 --> 00:01:06.633
The corrected SVG preserves one circle and changes its fill, so
the visible result now matches the required object.

11
00:01:07.613 --> 00:01:14.053
These are actual, simple SVG teaching examples, not borrowed
diagrams or measured language-model outputs.

12
00:01:14.753 --> 00:01:20.933
It shows why a visual output must be inspected for content after
its code passes the structural validity check.

13
00:01:21.553 --> 00:01:27.693
Generation starts from natural language, requiring the model to
construct the structured output it describes.

14
00:01:28.533 --> 00:01:34.693
Conversion starts from an existing structure, requiring
information and relationships to survive a format change.

15
00:01:35.673 --> 00:01:42.953
For our example, JSON containing blue and two can be converted
to YAML while preserving the same field meanings.

16
00:01:43.593 --> 00:01:50.333
The syntax changes, but a color value becoming red would be an
incorrect conversion even if the YAML remains parseable.

17
00:01:51.433 --> 00:01:56.573
Both task types appear in text-only outputs and in code that
produces rendered visual content.

18
00:01:57.313 --> 00:02:03.053
This separates constructing structure from preserving it across
formats, rather than treating every request alike.

19
00:02:03.740 --> 00:02:10.380
The overview states eighteen formats, forty-four task types, and
two thousand thirty-five curated examples.

20
00:02:11.380 --> 00:02:19.040
Text-only tasks include formats such as JSON, YAML, CSV, and
XML, which can be checked through their structured content.

21
00:02:20.100 --> 00:02:27.480
Renderable tasks include SVG, HTML, and other visual code, whose
displayed content must satisfy the specification.

22
00:02:28.780 --> 00:02:34.880
Generation and conversion in both subsets give four categories,
rather than one undifferentiated format score.

23
00:02:36.180 --> 00:02:42.320
Synthesized tasks and metrics receive two expert review passes
for clarity, consistency, and evaluability.

24
00:02:43.200 --> 00:02:48.920
The benchmark specifies required content, rather than merely
checking whether output brackets and tags are legal.

25
00:02:49.587 --> 00:02:56.907
A format-specific parser supplies a binary valid-or-invalid
syntax result for textual structured output.

26
00:02:57.627 --> 00:03:03.567
Keyword rules inspect names, values, and nested relationships
using paths or literal matching.

27
00:03:04.427 --> 00:03:10.447
A rule can inspect nested fields or list members, making
structure more specific than a visible-word collection.

28
00:03:11.347 --> 00:03:18.247
Generation requirements define rules; conversion rules specify
relationships that must survive format transformation.

29
00:03:19.087 --> 00:03:25.847
Text-only scores weight syntax at twenty percent and keyword
matching at eighty percent, combining complementary checks.

30
00:03:26.687 --> 00:03:32.027
A valid parse is useful, but required structural content
accounts for most of this task score.

31
00:03:32.460 --> 00:03:37.820
Visual code is rendered to check execution, and its produced
image supports further content checks.

32
00:03:38.740 --> 00:03:43.980
Keywords inspect code; a visual judge checks reference
question-answer pairs against the rendered view.

33
00:03:44.880 --> 00:03:50.420
The blue-circle example shows valid SVG displaying the wrong
color despite having the requested shape.

34
00:03:51.280 --> 00:03:57.680
The visual score weights syntax twenty percent, keywords ten
percent, and visual-question checks seventy percent.

35
00:03:58.700 --> 00:04:04.620
These are benchmark-specific checks, rather than a universal
certificate for every interaction or semantic property.

36
00:04:05.480 --> 00:04:11.720
Together they distinguish readable code, required structure, and
visible content, beyond successful execution alone.

37
00:04:12.313 --> 00:04:18.953
The pinned comparison uses uniform zero-shot prompts and greedy
decoding in the structural evaluation pipeline.

38
00:04:19.693 --> 00:04:25.813
Table six averages four categories: text and visual generation,
and text and visual conversion.

39
00:04:26.533 --> 00:04:30.713
The composite is seventy-six point zero two points for GPT four
o.

40
00:04:31.293 --> 00:04:36.273
It is sixty-seven point zero four points for Qwen version three,
with four billion parameters.

41
00:04:36.993 --> 00:04:43.233
These are weighted task scores, rather than the percentage of
outputs completely satisfying every required property.

42
00:04:44.133 --> 00:04:50.133
Category columns expose generation and conversion behavior; the
mean should not replace that detailed breakdown.

43
00:04:51.033 --> 00:04:56.153
The result belongs to this historical model set and benchmark
version, not every current language model.

44
00:04:56.787 --> 00:05:03.587
Both JSON and SVG can be syntactically valid while failing an
important requirement of the requested output.

45
00:05:04.287 --> 00:05:10.547
Generation constructs relationships; conversion preserves them
when representation changes to another format.

46
00:05:11.247 --> 00:05:17.967
Text checks inspect fields and paths; visual outputs also
require examining the content their code displays.

47
00:05:18.727 --> 00:05:25.087
StructEval separates syntax, structural keywords, and visual
checks, with subset-specific metric weights.

48
00:05:25.827 --> 00:05:31.567
Evaluate the artifact actually requested, rather than rewarding
a format that merely looks well formed.

49
00:05:32.267 --> 00:05:38.207
Useful output carries correct information in correct
relationships, and displays the right content when rendered.