1
00:00:00,340 --> 00:00:07,040
Suppose the request asks for structured data describing two blue
objects, with separate fields for color and count.

2
00:00:07,780 --> 00:00:13,480
Our first JSON object has valid punctuation and a count of two,
but its color value is red.

3
00:00:14,100 --> 00:00:20,640
A parser can read it successfully, so syntactic validity alone
would not reveal the mismatch with the requested content.

4
00:00:21,540 --> 00:00:26,680
Now compare a corrected object that keeps the same structure but
changes the color value to blue.

5
00:00:27,400 --> 00:00:33,520
Both objects are original teaching examples, not benchmark
submissions or measured outputs from an evaluated model.

6
00:00:34,340 --> 00:00:40,120
Readable format and required structure are different
achievements, motivating the distinctions measured by
StructEval.

7
00:00:40,613 --> 00:00:46,573
For a visual example, ask for one blue circle described by SVG
code that can be rendered.

8
00:00:47,253 --> 00:00:52,913
The candidate code is well formed and draws a circle, but the
fill color is red rather than blue.

9
00:00:53,773 --> 00:00:59,073
Successful rendering proves execution, while the displayed
circle still violates the requested color.

10
00:00:59,933 --> 00:01:06,633
The corrected SVG preserves one circle and changes its fill, so
the visible result now matches the required object.

11
00:01:07,613 --> 00:01:14,053
These are actual, simple SVG teaching examples, not borrowed
diagrams or measured language-model outputs.

12
00:01:14,753 --> 00:01:20,933
It shows why a visual output must be inspected for content after
its code passes the structural validity check.

13
00:01:21,553 --> 00:01:27,693
Generation starts from natural language, requiring the model to
construct the structured output it describes.

14
00:01:28,533 --> 00:01:34,693
Conversion starts from an existing structure, requiring
information and relationships to survive a format change.

15
00:01:35,673 --> 00:01:42,953
For our example, JSON containing blue and two can be converted
to YAML while preserving the same field meanings.

16
00:01:43,593 --> 00:01:50,333
The syntax changes, but a color value becoming red would be an
incorrect conversion even if the YAML remains parseable.

17
00:01:51,433 --> 00:01:56,573
Both task types appear in text-only outputs and in code that
produces rendered visual content.

18
00:01:57,313 --> 00:02:03,053
This separates constructing structure from preserving it across
formats, rather than treating every request alike.

19
00:02:03,740 --> 00:02:10,380
The overview states eighteen formats, forty-four task types, and
two thousand thirty-five curated examples.

20
00:02:11,380 --> 00:02:19,040
Text-only tasks include formats such as JSON, YAML, CSV, and
XML, which can be checked through their structured content.

21
00:02:20,100 --> 00:02:27,480
Renderable tasks include SVG, HTML, and other visual code, whose
displayed content must satisfy the specification.

22
00:02:28,780 --> 00:02:34,880
Generation and conversion in both subsets give four categories,
rather than one undifferentiated format score.

23
00:02:36,180 --> 00:02:42,320
Synthesized tasks and metrics receive two expert review passes
for clarity, consistency, and evaluability.

24
00:02:43,200 --> 00:02:48,920
The benchmark specifies required content, rather than merely
checking whether output brackets and tags are legal.

25
00:02:49,587 --> 00:02:56,907
A format-specific parser supplies a binary valid-or-invalid
syntax result for textual structured output.

26
00:02:57,627 --> 00:03:03,567
Keyword rules inspect names, values, and nested relationships
using paths or literal matching.

27
00:03:04,427 --> 00:03:10,447
A rule can inspect nested fields or list members, making
structure more specific than a visible-word collection.

28
00:03:11,347 --> 00:03:18,247
Generation requirements define rules; conversion rules specify
relationships that must survive format transformation.

29
00:03:19,087 --> 00:03:25,847
Text-only scores weight syntax at twenty percent and keyword
matching at eighty percent, combining complementary checks.

30
00:03:26,687 --> 00:03:32,027
A valid parse is useful, but required structural content
accounts for most of this task score.

31
00:03:32,460 --> 00:03:37,820
Visual code is rendered to check execution, and its produced
image supports further content checks.

32
00:03:38,740 --> 00:03:43,980
Keywords inspect code; a visual judge checks reference
question-answer pairs against the rendered view.

33
00:03:44,880 --> 00:03:50,420
The blue-circle example shows valid SVG displaying the wrong
color despite having the requested shape.

34
00:03:51,280 --> 00:03:57,680
The visual score weights syntax twenty percent, keywords ten
percent, and visual-question checks seventy percent.

35
00:03:58,700 --> 00:04:04,620
These are benchmark-specific checks, rather than a universal
certificate for every interaction or semantic property.

36
00:04:05,480 --> 00:04:11,720
Together they distinguish readable code, required structure, and
visible content, beyond successful execution alone.

37
00:04:12,313 --> 00:04:18,953
The pinned comparison uses uniform zero-shot prompts and greedy
decoding in the structural evaluation pipeline.

38
00:04:19,693 --> 00:04:25,813
Table six averages four categories: text and visual generation,
and text and visual conversion.

39
00:04:26,533 --> 00:04:30,713
The composite is seventy-six point zero two points for GPT four
o.

40
00:04:31,293 --> 00:04:36,273
It is sixty-seven point zero four points for Qwen version three,
with four billion parameters.

41
00:04:36,993 --> 00:04:43,233
These are weighted task scores, rather than the percentage of
outputs completely satisfying every required property.

42
00:04:44,133 --> 00:04:50,133
Category columns expose generation and conversion behavior; the
mean should not replace that detailed breakdown.

43
00:04:51,033 --> 00:04:56,153
The result belongs to this historical model set and benchmark
version, not every current language model.

44
00:04:56,787 --> 00:05:03,587
Both JSON and SVG can be syntactically valid while failing an
important requirement of the requested output.

45
00:05:04,287 --> 00:05:10,547
Generation constructs relationships; conversion preserves them
when representation changes to another format.

46
00:05:11,247 --> 00:05:17,967
Text checks inspect fields and paths; visual outputs also
require examining the content their code displays.

47
00:05:18,727 --> 00:05:25,087
StructEval separates syntax, structural keywords, and visual
checks, with subset-specific metric weights.

48
00:05:25,827 --> 00:05:31,567
Evaluate the artifact actually requested, rather than rewarding
a format that merely looks well formed.

49
00:05:32,267 --> 00:05:38,207
Useful output carries correct information in correct
relationships, and displays the right content when rendered.