Agent Evaluation Integrity: Turn Reward into an Evidence Protocol

An agent doing the right thing and an agent receiving a high score are not the same fact. A web toast, a 200 tool response, a higher test metric, a satisfied LLM judge, or even a hidden-verifier pass can cover only part of the goal. For agents that use tools, train, and edit artifacts, reward cannot be treated as the definition of completion. It should be a traceable, attackable evidence path that can be compared with an independent terminal state.

1. Separate Five Objects That Are Often Collapsed

Skalse et al. formally define reward gaming as improving a proxy reward while harming true reward, making the proxy-goal gap more than an implementation detail. Defining and Characterizing Reward Gaming further shows that for all stochastic policies, a nontrivial fully unhackable proxy is an exceptionally strong condition. The engineering implication is not to abandon reward, but not to mistake one scalar for complete semantics.

2. Five Integrity Failures, Five Different Controls

Reward Tampering Problems and Solutions causally distinguishes tampering with the reward function from tampering with reward-function inputs. For an agent, the practical audit question is whether a policy can change metric code, test data, trace collector, judge context, submission payload, or any artifact feeding the score without changing the postcondition the user cares about. The definition does not say every agent will tamper; it requires us to draw writable paths and evidence dependencies.

3. The Minimum Contract for Credible Evaluation

TaskSpec       = goal, initial_state_digest, policy_observations, authorized_actions, budget
ActionEvent    = actor, action, authority, timestamp, request/side_effect_digest
EvidenceEvent  = collector_identity, source, artifact/state_digest, visibility, append_only_pointer
RewardSignal   = version, inputs, aggregation, assignment_scope, training_use
Verifier       = version/hash, hidden_postcondition, inputs, diagnostics, execution_domain
Judge          = model/prompt/rubric/version, visible_evidence, calibration_protocol
Acceptance     = required_checks, integrity_constraints, cost/risk gate, decision, rollback

Seven engineering invariants follow:

1. The policy cannot write the verifier, hidden tests, metric code, judge prompt, or acceptance decision.
2. Reward inputs and final evidence are collected separately. The policy produces actions; a sidecar, backend, or independent runner observes results.
3. Every input carries a version and digest. A changed evaluator, judge prompt, or test dataset is a visible experimental variable.
4. Visibility is also authority. The policy cannot read hidden rules, reference answers, or privileged credentials through a page, tool response, memory, or error.
5. A process trace is valid only in its stated scope. It may explain, diagnose, or train, but does not by default replace a terminal postcondition.
6. The acceptance gate combines columns. verified_outcome, integrity, cost, authority violations, recovery, and human escalation are recorded separately rather than erased by one reward.
7. Rollback is outside candidate control. An evaluator loophole, contamination, or regression can be localized and accepted versions revoked.

4. Process Supervision Is Valuable, Not a License for Browser Clicks

Let’s Verify Step by Step compares process and outcome supervision on MATH reasoning and reports a benefit for its process-supervised reward model under its human step-level labels and task distribution. It supports making intermediate process a testable supervision object. It does not show that browser clicks, terminal commands, subagent messages, or natural-language rationales naturally make reliable dense rewards.

Before using a process signal in an agent environment, ask:

The minimum evidence for retaining a process reward is that, at equal budget on a frozen hidden terminal evaluator and adversarial tasks, it improves verified_outcome or lowers integrity/recovery risk. When process score rises while terminal state, authority, or safety worsens, it is another gameable proxy.

5. Credit Assignment Is Not Splitting Terminal Reward Across Every Token

A long-running agent session contains tool calls, failures, retries, context compression, parallel workers, and different authority domains. Broadcasting one terminal reward to every request or token can reward irrelevant or wrong behavior. Polar records token-level interaction through native-harness trajectory reconstruction and warns about coarse request-level reward broadcast. Agent Lightning proposes a hierarchical attribution interface that decomposes complex agent execution into training transitions. Both are designs for training interfaces and attribution, not validated dense-reward solutions for every browser or scientific environment.

When reporting credit assignment, state at least:

1. whether assignment scope is token, model request, tool action, subgoal, submission, or session;
2. whether each reward input comes from a policy-writable surface or independent sidecar;
3. how concurrent work, retries, failed branches, and cancellation are masked, penalized, or retained;
4. how all-zero/all-one groups, sparse returns, and related traces are handled; and
5. whether attribution was ablated on final holdouts, attack suites, and other harnesses.

Without this, the name PPO, GRPO, or any other update rule cannot say what was actually learned.

6. Make “Hackable” a First-Class Benchmark Outcome

Hack-Verifiable Environments embeds detectable reward-hacking opportunities directly in environments so exploitation is automatically verifiable. RewardHackingAgents separately measures evaluator tampering and train/test leakage in ML-engineering workspaces using patch tracking, file-access logging, and a trusted reference. These preprints support a practical design principle: do not wait for accidental cheating; include shortcuts an agent must not take, and score them separately.

Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops turns verifier hardening into a loop: a hacker finds an exploit that passes without solving, a fixer closes it, and a solver confirms legitimate solutions still pass. Its results are scoped to its terminal benchmarks, attack models, and patching process; they do not make a fixed verifier loophole-free. They do supply a key acceptance rule: after patching a loophole, also show that legitimate solutions are not rejected and attack unseen exploits again.

7. Cross-Environment Anti-Gaming Regressions

These regressions do not make reward perfect. They force a score to face the alternative explanation: did the agent achieve the goal, or merely reach a weak checkpoint?

8. Training, Evaluation, and the Core Five

Training may use reward, process signals, judges, or synthetic trajectories. A final research claim must state how they are isolated from frozen acceptance. For synthetic instances, the generator cannot see final verifier or successful evaluation traces. For RSI, the candidate cannot write acceptance. For Auto Research, prose quality cannot substitute for materials, interventions, and replayable evidence. For long horizons, self-reported success cannot replace backend state after recovery.

The conclusion is a modest engineering standard: reward is a challengeable training signal; a verifier is a versioned checker; a judge is an opinion with calibration error; and acceptance is an independent release or research decision. Separating them does not guarantee that an agent never finds a loophole. It makes loopholes, false blocks, leakage, and real completion different problems that can be measured, reproduced, and repaired.