Where, What, Why, and Importance: Structured Defect Grounding for Text-to-Image Feedback

Paper: Where, What, Why, and Importance: Structured Defect Grounding for Text-to-Image Feedback (v2) https://arxiv.org/html/2606.06113v2

Structured Defect Grounding represents image feedback as a set of localized defects. Each item combines a box, a top-level type, a reason and an importance score. The distinction between image-intrinsic artifacts and prompt-conditioned misalignments keeps the diagnosis tied to the appropriate evidence.

An original sign example asks for HELLO while displaying HELO. The inscription’s box tells an editor where to look; the missing-letter reason explains what conflicts with the prompt. The drawing is a teaching example rather than a measured detector or editor output.

The dataset, SDG-30K, contains 30,096 images from four generators. Its prompt-disjoint split has 28,945 training images and 1,151 test images. Boxes, types and short descriptions are combined with expanded training descriptions, reasoning traces and importance annotations.

A Qwen3-VL-4B detector uses supervised training followed by GRPO. A format gate checks structured JSON and valid geometry, while quality rewards address location, description consistency and importance. Evaluation separates image-level type presence/absence from defect-level boxes, descriptions and importance. A valid format alone does not establish a correct defect diagnosis.

On artifact-annotated test images, [email protected] is 0.200 for zero-shot Gemini3Pro, 0.255 for SDG-SFT and 0.263 for SDG-GRPO. The artifact-conditioned subset and metric belong with those values.

A separate refinement experiment keeps GPT-Image1.5 fixed and compares feedback treatments. Blinded paired judgments over 873 valid filtered samples give 10.31% Good, 86.94% Same and 2.75% Bad relative to fixed caption-only feedback. The companion animation follows how location, type, explanation and importance become editor inputs while preserving the measured outcome distribution, including its large unchanged fraction.

Pinned source versions: Paper, v2: https://arxiv.org/html/2606.06113v2

#ImageGeneration #VisualFeedback #AIResearch