1 00:00:00,260 --> 00:00:05,200 Imagine four cells in a row and a token that must reach the last cell. 2 00:00:06,000 --> 00:00:11,060 The rule is simple: each move must go to the adjacent cell, without skipping positions. 3 00:00:12,340 --> 00:00:18,200 One sequence visits every intermediate cell; another jumps directly from the start to the final cell. 4 00:00:19,040 --> 00:00:23,020 They finish in the same cell, but only one route respects the stated rule. 5 00:00:24,000 --> 00:00:28,740 These are original schematic sequences, not clips generated by the models evaluated in the paper. 6 00:00:29,720 --> 00:00:34,960 They show why a final picture alone cannot establish that a visual procedure was valid. 7 00:00:35,587 --> 00:00:43,467 VGI-Bench studies visual reasoning expressed through generated video, with tasks that require valid intermediate states. 8 00:00:44,287 --> 00:00:49,207 Each model receives an input image as the first frame and a text description of the task. 9 00:00:49,787 --> 00:00:56,447 The instructions define the goal, relevant objects, allowed actions, and shortcuts that must not be taken. 10 00:00:57,327 --> 00:01:02,507 The generated clip must make progress toward that goal while preserving the task's rules. 11 00:01:03,227 --> 00:01:09,727 A motionless clip can avoid many violations yet accomplish little; a jump can reach the endpoint incorrectly. 12 00:01:10,467 --> 00:01:16,227 The benchmark therefore measures both goal completion and the process that produced the apparent solution. 13 00:01:16,713 --> 00:01:21,573 The suite organizes twenty-seven task types into four mutually exclusive domains, 14 00:01:22,033 --> 00:01:23,973 according to their visual characteristics. 15 00:01:24,833 --> 00:01:28,313 Visual organization concerns arranging or selecting objects, 16 00:01:28,753 --> 00:01:32,833 while physical manipulation involves plausible object-level interaction. 17 00:01:33,513 --> 00:01:40,733 Structured puzzles require explicit rule-following, and spatiotemporal dynamics concern how states depend on time. 18 00:01:41,693 --> 00:01:44,313 Seven nonexclusive skill tags give another view, 19 00:01:44,613 --> 00:01:49,473 including planning, spatial reasoning, temporal reasoning, and topology. 20 00:01:50,153 --> 00:01:54,013 One task belongs to one domain but can require several skills at once. 21 00:01:54,873 --> 00:02:01,353 That separation helps diagnose whether a failure concerns the visual task family or a particular underlying skill. 22 00:02:02,020 --> 00:02:08,400 Tasks appear at three difficulty levels, with prompts and initial images reviewed for the intended procedure. 23 00:02:09,240 --> 00:02:15,160 The benchmark favors photorealistic-style inputs to reduce mismatch with video models' learned visual appearances. 24 00:02:16,140 --> 00:02:20,920 Before admitting a task, a small easiest-level sample is tried on several video models. 25 00:02:21,520 --> 00:02:24,860 At least one tested model must solve it, and at least one must fail. 26 00:02:26,040 --> 00:02:32,740 This screen avoids trivially solved or wholly infeasible proposals; it does not certify every instance's solvability. 27 00:02:33,840 --> 00:02:36,800 Our abstract four-cell drawing explains the rule distinction; 28 00:02:37,340 --> 00:02:39,980 it does not represent the benchmark's realistic input style. 29 00:02:40,427 --> 00:02:46,967 Completeness asks how far the video progresses, using task-specific complete, partial, or failed tiers. 30 00:02:47,707 --> 00:02:52,547 The tiers map to one, one-half, and zero in the reported evaluation formulation. 31 00:02:53,587 --> 00:02:59,187 A separate rubric checks local rules and counts violations, using focused inspection of relevant intervals. 32 00:03:00,047 --> 00:03:05,947 An item with more violations receives a smaller score; the rubric averages those item-level values. 33 00:03:07,007 --> 00:03:12,967 The final score multiplies completeness by rubric adherence, so either kind of failure reduces the result. 34 00:03:13,847 --> 00:03:19,527 This composite is different from strict success, which separately requires perfect completion and rubric scores. 35 00:03:20,147 --> 00:03:25,867 The main evaluation uses half the instances of each task, selected with a fixed random seed. 36 00:03:26,507 --> 00:03:31,747 A Gemini three Flash evaluator applies the completion and rubric procedure to generated clips. 37 00:03:32,667 --> 00:03:37,687 Seedance two point zero reports the highest overall score in the table, fifty-one point zero. 38 00:03:38,527 --> 00:03:44,667 Selected other rows report forty-four point zero for Kling three and thirty-six point seven for Sora two. 39 00:03:45,767 --> 00:03:49,647 These are composite benchmark scores under published generation configurations, 40 00:03:50,007 --> 00:03:53,067 not universal percentages of visual reasoning correctness. 41 00:03:54,087 --> 00:03:57,847 An image-output adaptation measures static target rendering separately, 42 00:03:58,207 --> 00:04:00,667 without certifying the intermediate video process. 43 00:04:01,267 --> 00:04:05,927 Another analysis studies decoded intermediate products during denoising, 44 00:04:06,307 --> 00:04:08,747 rather than events along the finished video's timeline. 45 00:04:09,587 --> 00:04:12,967 The experiment samples one hundred seventeen task instances 46 00:04:13,367 --> 00:04:16,927 and compares successive decoded checkpoints from tested open models. 47 00:04:17,787 --> 00:04:24,807 A stable state stays the same; self-correction means an identifiable wrong state becomes a correct state. 48 00:04:25,627 --> 00:04:29,867 Changing from one wrong solution to another wrong solution does not count as correction. 49 00:04:31,067 --> 00:04:37,987 In this sampled analysis, wrong-to-correct transitions remain at or below zero point nine percent per checkpoint pair. 50 00:04:38,927 --> 00:04:44,867 The finding describes these decoded-state experiments; it does not prove that all video models lack internal reasoning. 51 00:04:45,453 --> 00:04:52,193 Return to the token sequences: both finish in the same cell, but one skips a required intermediate state. 52 00:04:53,053 --> 00:04:57,773 That distinction remains visible only when we inspect the process, not just the last frame. 53 00:04:58,793 --> 00:05:05,133 A useful generated procedure must make progress while keeping the relevant rules and object states coherent. 54 00:05:06,033 --> 00:05:11,693 VGI-Bench makes those two obligations explicit through task design and its paired evaluation criteria. 55 00:05:12,453 --> 00:05:18,853 The goal is a diagnostic view of visual intelligence, rather than mistaking visual polish for a valid solution. 56 00:05:19,813 --> 00:05:24,313 Ask two questions of the clip: did it reach the goal, and was its route allowed?