Agentic RL: The Harness Is Part of the Policy

Conventional RL often imagines a tidy API: observations arrive, actions leave, and rewards return. Real coding agents do not run that way. They have tool protocols, context compression, subagents, terminals, retries, and stopping rules. For agentic RL, this agent harness is not disposable packaging. It is part of the condition under which the policy actually executes.

1. Why Forcing an Agent into an Ordinary Environment API Loses Information

A mature harness defines how system prompts are assembled, how tools return observations, when context is compressed, how subagents run concurrently, how files persist, and when execution stops. Rewriting it as an environment owned by a training framework can alter those behaviors or lose tokens, log-probabilities, branches, and tool context.

This is the problem addressed by Polar. It is neither a new PPO / GRPO objective nor a benchmark. It is an asynchronous rollout system: the native harness points model requests to a gateway, the gateway forwards and records the interactions, and training trajectories are reconstructed from the real execution.

2. Polar’s System Boundary

native harness
  -> model API proxy
  -> inference backend
  -> CompletionSession
  -> trajectory builder
  -> evaluator / reward
  -> independent trainer

• The harness stays native. The paper supports several provider-shaped model APIs and captures prompts, sampled tokens, token IDs, and log-probabilities at the proxy.
• A trajectory is not a pretty transcript. Strict prefix continuations may be merged. Subagents, parallel branches, and context rewrites should remain separate chains rather than be forced into one linear conversation.
• Evaluation stays separate. Tests, session completion, or other evaluators generate outcome or process rewards after rollout; the trainer need not own the harness.

Four layers should be recorded independently: runtime isolation, harness launch configuration, model traffic and trajectory reconstruction, and verification plus reward. Otherwise a score change cannot be attributed to the model, harness, environment, or grader.

3. What the Paper Actually Shows

In the paper’s specified setup, the authors start from the same Qwen3.5-4B base, use standard GRPO and 293 SWE-Gym training tasks, and report the following absolute pass@1 changes on SWE-Bench Verified across four coding harnesses:

The authors also report that prefix_merging reduced trainer-side updates from 1,185 to 218 and shortened wall-clock time in one reconstruction ablation. This is evidence about the paper’s design and stated experimental setting, not a law that every harness will receive equal gains.

4. Reward Attribution Is the Dangerous Part

The final patch of a long-running agent may pass a verifier, but that does not make every model call equally responsible. Polar explicitly finds that mechanically broadcasting a session outcome reward to every request-level trace creates a substantial reward-hacking risk.

A credible agentic RL environment should therefore disclose:

1. whether reward is assigned to the final artifact, a submission, a step, or the full session;
2. how reward maps onto token-level loss masks;
3. whether tool failures, retries, subagent branches, and context compression remain in the trajectory;
4. where the verifier runs and whether the agent can contaminate it; and
5. whether training, validation, and deployment use the same harness version.

5. How This Connects to Environment Design

The benchmark, harness, and sandbox layers in the environment post still apply to agentic RL. A fourth layer must be added: the trajectory and training interface. Evaluation decides what completion means. The harness determines how the policy acts. The sandbox determines visible state and permissions. The training interface determines which behavior receives credit, optimization, and replay.

That is why rerunning the same model with another harness is not an incidental engineering detail. Prompts, tool availability, stopping policy, context policy, and evaluation can all change the rollout distribution. A training result should name those conditions rather than present a final score as an environment-free model property.

6. A Checklist for Personal Experiments

1. Fix and record harness, model endpoint, system prompt, tool versions, sandbox image, and verifier hash.
2. Save replayable token-aligned trajectories, not only natural-language transcripts.
3. Separate training reward from independent final evaluation and report their gap.
4. Evaluate on at least one task distribution that was not selected or tuned against, so harness adaptation is not reported as general ability.