VideoScore2: Think before You Score in Generative Video Evaluation

Paper: VideoScore2: Think before You Score in Generative Video Evaluation (v1) https://arxiv.org/html/2509.22799v1

This explainer follows the mechanism before interpreting its reported results.

A generated video can be sharp yet show the wrong object, or match a prompt while breaking physical continuity. VideoScore2 evaluates visual quality, text alignment and physical/common-sense consistency. These are distinct questions, but a single defect can affect more than one dimension.

Our original blue-ball animation illustrates blur, a color mismatch and a sudden jump. The examples are teaching graphics, not research videos or measured critic predictions. They make the evaluation questions concrete without claiming controlled single-dimension interventions.

The training data contains 27,168 videos and 81,504 dimension scores. A 500-video set is held out as VideoScoreBench-v2. Human one-to-five ratings and short comments are expanded and reconciled through an LLM-assisted rationale pipeline. The detailed training rationales therefore do not all come directly from human annotation.

The critic starts from Qwen2.5-VL-7B-Instruct. Supervised fine-tuning establishes the task, and GRPO refines score agreement. The reported reinforcement-learning checkpoint uses 300 steps, with two frames per second at inference. The method returns rationales and three dimension scores rather than a single unqualified judgement.

In-domain exact mean accuracy is 44.35, compared with 38.41 for VideoScore 1.1. Relaxed agreement is 90.78 and allows a difference of one score point; it should not be substituted for exact accuracy. The average across four out-of-domain evaluation columns is 50.37, versus 46.05 for Q-Insight. Those columns mix preference and point-score metrics, so the average is not universal accuracy or evidence of winning every benchmark.

A separate best-of-five experiment selects from existing candidates for 500 prompts and six moderate or poor generators, comparing selection with random choice. That experiment evaluates a critic’s usefulness for selection; it does not train generator weights during that selection process.

The central idea is to connect a score to a diagnostic explanation and to keep the metric’s meaning visible when comparing results.