RewardHarness: Learning Human Preferences for Image Editing with Only 100 Demonstrations

Paper: RewardHarness: Learning Human Preferences for Image Editing with Only 100 Demonstrations (COLM 2026) https://openreview.net/forum?id=tXv4YSeA2j

Paper: RewardHarness: Self-Evolving Agentic Post-Training (Related earlier version, arXiv v1) https://arxiv.org/abs/2605.08703v1

COLM 2026 is the conference version; arXiv v1 is an earlier version of the same work.

This explainer follows the mechanism before interpreting its reported results.

Suppose an image-edit instruction asks for a red cup while keeping a sign unchanged. One candidate recolors the cup but drops a letter from the sign. Another preserves the text. The first edit satisfies a visible part of the request and still violates the whole instruction. Our original vector example makes that evaluation problem concrete; it is not a research image or measured result.

The study keeps the evaluator model fixed and evolves its written context. Skills provide declarative rubrics and examples. Tools provide targeted visual-analysis procedures that the vision-language model follows; they are not a claim that an external executable OCR engine is being invoked. An orchestrator selects relevant documents, applies criteria, optionally follows checks, and aggregates a preference ranking.

A small calibration set supplies the improvement signal. The 100 demonstrations are divided approximately into 70 training examples and 30 held-out validation examples. Candidate document-library changes are accepted only when validation strictly improves. Otherwise the incumbent library is restored. The final library is frozen before the separate evaluation.

Table 1 compares contexts using the same Qwen2.5-VL-7B-Instruct backbone: raw evaluation averages 30.3%, EditReward 42.0%, and the learned context 45.7%. These are means across four evaluation columns, not pooled accuracy. A separately evaluated Gemini 2.0 Flash variant reaches 47.4%; it is a different evaluator configuration.

The downstream experiment asks another question: can this reward guide editor training? With FlowGRPO and FLUX.2-klein-base-4B, the ImgEdit overall scores are 3.32 for the base editor, 3.45 with EditReward training, and 3.52 with the learned reward. Those values are points, not percentages.

The mechanism is improvement through explicit evaluator context: write down what matters, validate changes, and retain only changes that earn their place.