Coding Agent 架构对比 Coding Agents — Architecture Comparison

Jul 2026 · Harness Engineering / Long-Horizon Agents / Evaluation

A long-running agent is not "a model plus a prompt." It is a runtime system: it selects information within a finite context, calls tools, preserves artifacts, enforces authority, manages budget and recovery, and hands results to an evaluator the agent cannot rewrite. Harness engineering is the discipline of designing that layer.

Central claim: a trustworthy agent result belongs to a configuration, not only to a model: model + harness version + environment + tool/permission policy + evaluator + budget. A pass-rate increase alone therefore cannot show that "the model got better," nor that a more elaborate orchestration helped.

10.1 What This Section Does Not Do

This is not a product survey, a declaration that more subagents are always better, or a claim that RSI has arrived. The six deep dives above explain how six concrete systems run. This section asks a different question: how should a harness itself be designed, constrained, and evaluated?

Two meanings must remain separate:

1. An agent harness is the runtime around a model: context, tools, state, control flow, authority, logs, and recovery.
2. An evaluation harness provisions a task, executes trials, collects evidence, and scores results. The evaluated agent must not treat it as an editable tool.

When those meanings collapse, "the agent's score improved" becomes dangerously ambiguous. A reader cannot tell whether model, prompt, tools, environment, budget, or scorer changed.

10.2 Draw an Accountable Boundary First

An evidence rail outside the agent's write boundary: version manifests, permission decisions, tool events, artifact hashes, usage, terminal state, and evaluator outputs. Humans approve high-risk escalations or interpret uncertain conclusions at explicit handoff points.

This is not a diagram of one released system. It is a design position: the model may propose actions and the harness may schedule them, but authority and acceptance criteria cannot live only in a prompt that says "be careful." MCP provides a protocol boundary for objects such as tools, resources, and prompts. It does not automatically make a tool response trustworthy or replace host-side policy enforcement. MCP Specification

10.3 Six Engineering Planes

3.1 Contract: Tools Are Not Just a List of Function Names

Tools are contracts between a stochastic agent and stateful services. An evaluable tool surface makes its input schema, authority, errors, idempotency, and response scope inspectable. The more overlapping tool paths an agent has, the easier it is to obtain a fragile result through an accidental route. Anthropic's tool-design guidance recommends testing tools on realistic multi-step tasks and using held-out tasks to detect overfitting to development trajectories. Writing tools for agents

The testable question is not "how many tools exist?" It is whether task-shaped narrow tools reduce invalid calls, token waste, or policy violations at equal authority and budget without harming held-out success.

3.2 State: Turn Memory into Artifacts, Not an Ever-Longer Prompt

Context engineering is the selection and maintenance of token state visible at each invocation. It is not merely a larger context window, and it is not dumping every past log into a prompt. Long-running systems need task manifests, progress, tests, candidate artifacts, dependencies, and version history as retrievable external artifacts. A fresh session can then continue from the same evidence instead of guessing what a prior session did.

Three terms deserve a strict distinction:

Public long-running-harness practice emphasizes feature lists, progress files, tests, bootstrap commands, and version history so a new worker reconstructs state from artifacts rather than hidden chat memory. Effective harnesses for long-running agents

3.3 Authority: Runtime Enforcement, Not Prompt Wording

A prompt can express a preference; it cannot enforce an authority boundary by itself. A serious policy names readable and writable mounts, allowed hosts, credential scope, risky action classes, approval points, resource ceilings, network egress, and audit retention. Sandboxing and network controls are runtime mechanisms, not courtesy instructions. This is why public sandboxing designs discuss both filesystem and network boundaries. Claude Code sandboxing

One simple rule follows: an agent may propose a policy expansion, but it cannot approve that expansion itself, nor rewrite the evaluator and announce that it passed. The rule applies to browser, Computer Use, terminal, and training harnesses alike.

3.4 Control: Parallelism Is a Scheduling Hypothesis

Parallel subagents can widen research and independent execution coverage, but also multiply token, coordination, and merge cost. They fit only materially separable work with stable output contracts; shared code surfaces and tightly coupled decisions often need a common controller. A public multi-agent research system likewise reports substantial token cost for broader exploration and stresses that coordination shapes the outcome. Multi-agent research system

The parent should own global budget, decomposition, cancellation, merge, and publication. Workers should receive narrow objectives, isolated workspaces where possible, explicit deliverables, and durable result records. "Multi-agent is better" becomes an empirical claim only after comparison with a same-model, same-budget, same-task sequential baseline.

3.5 Evidence: Trace, Replay, and Rerun Are Not Synonyms

Every run should at least record harness, model, tool, environment, and policy versions; an initial-state digest; input/output and tool events; approvals and denials; artifact hashes; retries and recovery; evaluator outputs; and token, wall-clock, and resource usage. Secrets, personal data, and third-party content need redaction before retention.

That trace supports three different operations:

1. Diagnosis: why a run failed, spent unusually, or touched authority.
2. Observational replay: inspect an event record without repeating external side effects.
3. Execution rerun: execute from frozen state while declaring nondeterminism and external-dependency controls.

A video or chat transcript generally supports only the first two. It should not be called a reproducible rerun of a changing online service.

3.6 Integrity: The Evaluator Is Part of the System but Not the Optimization Surface

Deterministic tests, model graders, and human review answer different questions. Deterministic verifiers check explicit terminal states; model graders can handle nuanced text but need calibration and error reporting; humans catch unmodeled harm. Agent-evaluation engineering guidance stresses defining tasks, graders, samples, and metrics before interpreting a result as capability. Demystifying evals for AI agents

Harness changes should therefore carry a change manifest: what should improve, what must not regress, the held-out split that accepts it, and the rollback condition. Harness-Bench and VeRO are preprints exploring configuration-level or versioned agent evaluation. They provide useful methodological clues, not proof that elaborate harnesses are stronger on every task or evidence of unbounded self-improvement. Harness-Bench · VeRO

10.4 What a Harness Report Must Include

This table turns "the harness is good" into claims that can fail. A higher pass rate may come from hidden retries, greater concurrency, or an unequal token budget; omitting those makes the result unhelpful for engineering decisions.

10.5 When a Harness Tries to Improve Itself

Harness optimization can be useful: a system analyzes failed traces, proposes a small context, tool, or control-flow edit, tests it on frozen evaluation, then retains or reverts the version. It is still not automatically RSI. A stronger claim would require the modified harness to make later improvement processes reliably better on tasks that did not select it, while authority and evaluation remain outside the candidate's control.

The minimum acceptable loop is:

1. A human or external policy freezes editable surface, budget, evaluator, and rollback rule.
2. The agent proposes a narrow edit from failure lineage and declares expected gain and regression risk.
3. An independent runner executes capability and regression splits under the same budget and records evidence.
4. An external acceptance gate merges, rejects, or reverts; the candidate harness has no write access to that gate.

This matches the boundary in the RSI note: improving one workflow is valuable, but one local gain does not establish that the system improved "the improver."

10.6 A Workshop Agenda That Can Be Discussed Publicly

The following is a research agenda, not an announced event, CFP, or sponsorship notice.

1. Interfaces and environments. Tool protocols, browser and computer actions, sandboxing, portability, and deterministic verification.
2. Memory and long-horizon execution. Context lifecycles, artifact stores, checkpoints, recovery, durable jobs, and replay.
3. Governed autonomy. Capability scoping, identity, approval design, prompt-injection containment, and audit logs.
4. Optimization and evaluation. Workflow search, multi-agent scheduling, harness evolution, trace analysis, evaluator integrity, cost, and negative results.

A systems paper that claims a harness improvement should at minimum provide a versioned harness/environment manifest, task and evaluator versions, a budget ledger, trace/redaction plan, same-budget baseline, critical ablation, threat model, and failure analysis. These requirements do not guarantee correctness, but they expose a paper's load-bearing dependencies to readers.

10.7 Five Falsifiable Research Questions

These questions join AutoResearch, Agentic RL (Section 11), Agent Research Environments, and Agent Planning: all require separate boundaries for what can change, what feedback is observed, and what finally counts as evidence; the planning guide further separates plan, replanning, refusal, and terminal acceptance.

10.8 Primary References

• MCP Specification.
• Anthropic: Writing tools for agents, effective harnesses for long-running agents, multi-agent research system, sandboxing, and agent evaluation.
• Harness-Bench and VeRO (preprints).
• Lilian Weng, Harness Engineering for Self-Improvement.
• Related notes in this series: coding-agent architecture (above), research evidence map, RSI.

Jul 2026 · Agentic RL / Polar / RL Environments

Section 10 treated the harness as something to design and hold accountable; this section turns to training. Conventional RL often imagines a tidy API: observations arrive, actions leave, and rewards return. Real coding agents do not run that way. They have tool protocols, context compression, subagents, terminals, retries, and stopping rules. For agentic RL, this agent harness is not disposable packaging. It is part of the condition under which the policy actually executes.

Central claim: if training changes the behavior of the test-time harness, it is not training the policy that will run at deployment. Polar is valuable because it moves the rollout integration point to the model API boundary while leaving the native harness as intact as possible.

11.1 Why Forcing an Agent into an Ordinary Environment API Loses Information

A mature harness defines how system prompts are assembled, how tools return observations, when context is compressed, how subagents run concurrently, how files persist, and when execution stops. Rewriting it as an environment owned by a training framework can alter those behaviors or lose tokens, log-probabilities, branches, and tool context.

This is the problem addressed by Polar. It is neither a new PPO / GRPO objective nor a benchmark. It is an asynchronous rollout system: the native harness points model requests to a gateway, the gateway forwards and records the interactions, and training trajectories are reconstructed from the real execution.

11.2 Polar's System Boundary

native harness
  -> model API proxy
  -> inference backend
  -> CompletionSession
  -> trajectory builder
  -> evaluator / reward
  -> independent trainer

• The harness stays native. The paper supports several provider-shaped model APIs and captures prompts, sampled tokens, token IDs, and log-probabilities at the proxy.
• A trajectory is not a pretty transcript. Strict prefix continuations may be merged. Subagents, parallel branches, and context rewrites should remain separate chains rather than be forced into one linear conversation.
• Evaluation stays separate. Tests, session completion, or other evaluators generate outcome or process rewards after rollout; the trainer need not own the harness.

Four layers should be recorded independently: runtime isolation, harness launch configuration, model traffic and trajectory reconstruction, and verification plus reward. Otherwise a score change cannot be attributed to the model, harness, environment, or grader.

11.3 What the Paper Actually Shows

In the paper's specified setup, the authors start from the same Qwen3.5-4B base, use standard GRPO and 293 SWE-Gym training tasks, and report the following absolute pass@1 changes on SWE-Bench Verified across four coding harnesses:

The authors also report that prefix_merging reduced trainer-side updates from 1,185 to 218 and shortened wall-clock time in one reconstruction ablation. This is evidence about the paper's design and stated experimental setting, not a law that every harness will receive equal gains.

11.4 Reward Attribution Is the Dangerous Part

The final patch of a long-running agent may pass a verifier, but that does not make every model call equally responsible. Polar explicitly finds that mechanically broadcasting a session outcome reward to every request-level trace creates a substantial reward-hacking risk.

A credible agentic RL environment should therefore disclose:

1. whether reward is assigned to the final artifact, a submission, a step, or the full session;
2. how reward maps onto token-level loss masks;
3. whether tool failures, retries, subagent branches, and context compression remain in the trajectory;
4. where the verifier runs and whether the agent can contaminate it; and
5. whether training, validation, and deployment use the same harness version.

A Reward Signal Cannot Substitute for Outcome Fields

Reward Hacking Benchmark (RHB) places shortcuts such as skipped verification, task-adjacent metadata leakage, and evaluation-relevant tampering in multi-step tool tasks, and separates independent from chained regimes. Hack-Verifiable Environments instead embeds detectable vulnerability opportunities directly in an environment. Together they support treating exploitation as a rollout outcome rather than an after-the-fact guess. Both are scoped to their stated tasks and attack surfaces; RHB's model, post-training, and hardening comparisons are not a general ranking of PPO, GRPO, SFT+RL, or a guard.

Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops adds a necessary check: after patching a verifier, legitimate solutions still need to pass, and unseen exploits should be attacked again. Its terminal-benchmark results are likewise evidence under a stated process, not a safety proof for every agentic-RL environment.

Evidence boundary: "Any harness" means a harness that can be redirected at a model API boundary supported by the proxy. The main evidence covers software engineering, one 4B base model, four coding harnesses, and a specified training set. It does not directly establish training gains for browser or GUI environments.

11.5 Sources Answer Different Questions

Agentic RL can easily compress "it runs RL," "it has trajectories," "it has a verifier," and "training transfers" into one sentence. The following sources matter, but their evidence scopes differ:

The map blocks two common misreadings. First, GUI trajectory data and online RL rollouts are not the same object: the former can support SFT, offline filtering, or training-data research; the latter also needs a behavior policy, reward, attribution, and update rule. Second, possessing a verifier does not mean that it is isolated from the agent. Harbor distinguishes shared from separate verifiers; the task author still has to declare visible artifacts, network, and authority boundaries.

11.6 How This Connects to Environment Design

The benchmark, harness, and sandbox layers in the environment post still apply to agentic RL. A fourth layer must be added: the trajectory and training interface. Evaluation decides what completion means. The harness determines how the policy acts. The sandbox determines visible state and permissions. The training interface determines which behavior receives credit, optimization, and replay.

That is why rerunning the same model with another harness is not an incidental engineering detail. Prompts, tool availability, stopping policy, context policy, and evaluation can all change the rollout distribution. A training result should name those conditions rather than present a final score as an environment-free model property.

11.7 PPO, GRPO, and SFT+RL: Align the Rollout Ledger Before Comparing Update Rules

An algorithm label is not an experimental condition. Even if two runs both say "GRPO," they are not an interpretable comparison when the base checkpoint, harness, reset, tool authority, rollout stopping policy, reward, effective tokens, or evaluation selection differ. For long-running agents, those differences can be larger than the optimizer label.

The recent Single-Rollout Asynchronous Optimization (SAO) makes the issue concrete. It argues that GRPO's group-wise sampling does not naturally fit asynchronous agent rollouts, and reports results for a single-rollout asynchronous update in its coding and reasoning settings. That is a systems hypothesis worth testing, not a universal ranking of PPO, GRPO, or any harness. SAO is a preprint submitted on 2026-07-08; a comparison should disclose how groups are formed, behavior-policy lag, which rollouts actually enter an update, and resource use.

Freeze the minimum information in three manifests so a result can be reviewed rather than reduced to a training curve:

RolloutManifest(
  task_and_reset, base_and_harness, action_surface,
  behavior_checkpoint, termination_and_retry_policy,
  rollout_tokens, tool_calls, environment_resets, wall_clock, cost
)

OptimizationManifest(
  method, reward_and_verifier_version,
  ppo_value_reference_config_or_grpo_group_rule,
  sft_data_manifest_and_checkpoint, update_schedule, policy_lag
)

DeploymentAcceptance(
  run_hash, final_harness, independent_suite,
  proxy_reward, verified_outcome, exploit_outcome, hardening_delta,
  integrity, recovery, cost, transfer, decision
)

This is not ceremonial extra metadata. The current TRL GRPO documentation exposes a stateful environment_factory, reset, environment-owned reward, and a tool loop, and requires a prefix-preserving chat template for that loop. Those are concrete system conditions. Likewise, the TRL PPO interface names a policy, reference, reward, and value model. A framework exposes an interface; it does not automatically make an environment realistic, a reward safe, or transfer held out.

A falsifiable comparison can proceed as follows:

1. Freeze the RolloutManifest and train/validation/final split, and state which algorithm layer alone may change.
2. With the same base, harness, action surface, reward/verifier, and total token/rollout/reset/wall-clock budget, construct baseline, SFT-only, PPO, GRPO, or SFT+RL conditions. When resources are tight, compare fewer conditions rather than silently changing budgets.
3. Beyond mean reward, report the task/workflow distribution, failure branches, effective-rollout rate, GRPO group health, policy lag, verified_outcome, exploit_outcome, hardening's legitimate-pass/false-block behavior, and cost per verified success.
4. Run DeploymentAcceptance on a final suite that did not select checkpoints, prompts, harnesses, or hyperparameters. Rising training reward or a pass rate in the training environment cannot substitute for it.

Research wording: "PPO may outperform GRPO" or "PPO is better for ClawBench V2" is presently a hypothesis to test. Only results with a shared rollout ledger, independent acceptance, and task-distribution breakdown earn the status of a conditional conclusion.

11.8 A Checklist for Personal Experiments

1. Fix and record harness, model endpoint, system prompt, tool versions, sandbox image, and verifier hash.
2. Save replayable token-aligned trajectories, not only natural-language transcripts.
3. Separate proxy_reward, verified_outcome, exploit_outcome, and hardening_delta; training reward cannot substitute for an independent terminal or exploit report.
4. Evaluate on at least one task distribution that was not selected or tuned against, so harness adaptation is not reported as general ability.
5. For PPO, GRPO, and SFT+RL, publish a shared rollout, optimization, and acceptance manifest first; name groups, policy lag, checkpoint selection, and every budget.
6. Label each piece of evidence as a native rollout, offline trajectory, task/verifier contract, or algorithm objective; do not transfer a conclusion across those categories by default.

For separating reward, evidence, verifier, judge, and final acceptance, and for turning tampering, leakage, trace integrity, and process-reward gaps into adversarial regressions, see Agent Evaluation Integrity: Turn Reward into an Evidence Protocol. For the objective-level PPO / DPO / GRPO details, see Section 12 below.

11.9 References

• Xu et al., Polar: Agentic RL on Any Harness at Scale (preprint).
• Polar HTML paper, especially Sections 3 and 4.
• Luo et al., Agent Lightning (preprint); Harbor task documentation; and TRL's Harbor integration.
• Shao et al., DeepSeekMath / GRPO; Hou et al., Single-Rollout Asynchronous Optimization (preprint); Xu et al., AgentTrek.
• Reward Hacking Benchmark (ICML 2026; task-protocol-scoped multi-step shortcut measurement), Hack-Verifiable Environments, and Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops.
• TRL GRPO Trainer and TRL PPO Trainer, checked 2026-07-13; framework documentation, not transfer evidence.
• PPO / DPO / GRPO (Section 12) for the objective-level view of LLM RL.
• Agent research environments for task, harness, sandbox, and verifier boundaries.

Apr 2026 · RLHF / Preference Optimization / Reasoning RL

Section 11 covered how rollouts are collected from a native harness; this section turns to the objectives that consume them. It focuses on three central algorithms: PPO as the classic online RLHF workhorse, DPO as a common offline preference optimization method, and GRPO as a critic-free RL method often used in reasoning-model training. Many variants deserve their own post; this section sticks to the core ideas.

One sentence: PPO and GRPO are both policy-gradient-style online RL methods: the model samples answers first, then updates generation probabilities using rewards. DPO rewrites the KL-regularized RLHF objective into an offline preference-classification loss. All three use KL, a reference model, or clipping to limit policy drift.

12.0 Memorize This Table First

Mnemonic:

PPO  = online sampling + reward model/function + critic baseline + KL/clip
DPO  = offline preference pairs + reference model + classification-style loss
GRPO = online multi-sampling per prompt + group-relative reward + no critic

12.1 Shared Foundation: Policy Gradient

An LLM can be viewed as a policy:

pi_theta(y | x)

Given a prompt x, the model generates an answer y. The RL objective is to maximize expected reward:

maximize_theta E_{x ~ D, y ~ pi_theta(. | x)}[R(x, y)]

The simplest update direction is:

grad J(theta) ~= A(x, y) * grad log pi_theta(y | x)

Intuition:

• If an answer is better than the baseline, increase its generation probability.
• If an answer is worse than the baseline, decrease its generation probability.
• Real training must reduce variance, control KL drift, and handle long sequences, which is why more stable objectives such as PPO and GRPO are used.

12.2 PPO: Classic RLHF

The standard PPO-based RLHF pipeline is:

SFT model
-> train reward model from preference data
-> online rollout with current policy
-> PPO update policy under KL constraint

A common LLM PPO reward is:

reward = reward_model_score - beta * KL(pi_theta || pi_ref)

PPO training usually has four roles:

The core PPO objective can be understood as:

ratio = pi_theta(token) / pi_old(token)
objective = min(ratio * advantage, clip(ratio, 1-eps, 1+eps) * advantage)

Here pi_old is the old policy that produced the rollout samples. During the PPO update, the importance ratio compares the current policy with the policy used at sampling time. The reason for clip is simple: if the new policy changes too much relative to the old policy, PPO truncates the update to avoid destabilizing training in a single gradient step.

The PPO Workflow

Suppose the prompt is:

Write a Python function that checks whether a bracket string is valid.

The current policy samples this answer:

Scan the string with a stack; push left brackets, and when a right bracket appears, check whether it matches the stack top.

Training then goes through these steps:

A numeric example:

reward_model_score = 0.86
kl_penalty = 0.06
actual_return = 0.80

critic prediction at this prefix = 0.55
advantage = 0.80 - 0.55 = +0.25

This means the generation was better than the critic expected, so PPO increases the probability of the relevant tokens in that answer. But if a token probability changes too aggressively:

ratio = pi_theta(token) / pi_old(token) = 1.35
clip range = [0.8, 1.2]

the surrogate objective for that sample is clipped to 1.2 * advantage, so it no longer encourages the ratio to grow further. The intuition is:

Move toward higher reward, but do not take too large a step at once.

If another answer has low reward:

actual_return = 0.20
critic prediction = 0.55
advantage = -0.35

the corresponding token probabilities are pushed down. PPO is not just asking "what is the reward?" It is asking:

Was this generation better than the critic expected?
Did the new policy change too much relative to the old policy?
Is the new policy too far from the reference model?

PPO strengths:

• It is online RL and can directly optimize a reward signal.
• It is suitable for general RLHF when the reward model is reasonably reliable.
• With a critic and clipping, it is more stable than raw REINFORCE.

PPO weaknesses:

• It is engineering-heavy: policy / reference / reward / critic models must work together.
• Unstable critic training directly corrupts the advantage estimates.
• It is sensitive to hyperparameters, KL coefficients, and reward scale.

12.3 DPO: Preference Optimization, Not Traditional Online RL

DPO directly consumes preference pairs:

prompt x
chosen answer y_w > rejected answer y_l

It does not require training an explicit reward model first, and it does not need online rollout. The core idea is to make the model prefer the chosen answer over the rejected answer relative to a reference model:

log pi_theta(y_w | x) - log pi_ref(y_w | x)
should be larger than
log pi_theta(y_l | x) - log pi_ref(y_l | x)

An intuitive DPO loss:

loss = -log sigmoid(
  beta * [
    log_ratio(chosen) - log_ratio(rejected)
  ]
)

where log_ratio(y)=log pi_theta(y|x)-log pi_ref(y|x), and beta controls preference strength and conservatism relative to the reference. The reference model is still crucial: it is not a critic; it is the behavioral anchor that limits how far the model moves away from the original policy.

Technically, DPO is best described as an offline preference optimization method derived from KL-regularized RLHF. It uses the closed-form relationship between an optimal KL-regularized policy and its implicit reward to avoid training a separate reward model and running online RL.

How DPO Works

DPO does not let the model learn by online trial and error. It trains directly on preference data. For example:

Prompt:
Explain the difference between a reward model and a critic.

Chosen:
A reward model is a judge that scores a complete answer; a critic is a predictor that estimates how much future reward the current state can obtain.

Rejected:
A reward model and a critic both score the model; there is not much difference.

DPO asks whether the current model, relative to the reference model, prefers the chosen answer.

Suppose the current policy log probabilities are:

log pi_theta(chosen | x)   = -120
log pi_theta(rejected | x) = -115

This means the current model is actually more likely to generate the rejected answer, because -115 is a higher log probability than -120.

Now look at the reference model:

log pi_ref(chosen | x)   = -118
log pi_ref(rejected | x) = -116

DPO compares the two reference-relative ratios:

chosen_log_ratio   = -120 - (-118) = -2
rejected_log_ratio = -115 - (-116) = +1
gap = chosen_log_ratio - rejected_log_ratio = -3

The gap is negative, so the current model is more tilted toward the rejected answer than the reference is. The DPO loss pushes it to:

increase log pi_theta(chosen | x)
decrease log pi_theta(rejected | x)

So the training signal is not:

this answer has reward = 0.8

It is:

for the same prompt, the chosen answer should improve relative to the reference more than the rejected answer

In practice, implementations usually sum or average token log probabilities across the whole answer. Answer length, truncation, and tokenizer behavior all affect the values, so chosen / rejected samples should be comparable under the same prompt.

DPO's main advantage is engineering simplicity: no rollout, no explicit / separate online reward-model scoring, and no critic. Its limitation comes from the same place: the model learns only from existing preference pairs and does not actively explore new, better answers.

DPO strengths:

• Simple and stable; no online RL environment is required.
• No separate reward model or critic is needed.
• Very convenient when chosen / rejected preference pairs already exist.

DPO weaknesses:

• It does not actively explore new answers; it learns from offline preference data.
• For math and code tasks with verifiable outcomes, direct online RLVR is often stronger.
• If preference coverage is poor, the model may learn style more than capability.

12.4 GRPO: Critic-Free RL Often Used for Reasoning Models

The core idea of GRPO is to sample multiple answers for the same prompt and compare them within the group.

Prompt x
├── answer A -> reward 0.9
├── answer B -> reward 0.7
├── answer C -> reward 0.2
└── answer D -> reward 0.0

Use the group mean as the baseline and the group standard deviation for normalization:

mean = 0.45
std  = 0.36
A_adv = (0.9 - 0.45) / 0.36 = +1.25
B_adv = (0.7 - 0.45) / 0.36 = +0.69
C_adv = (0.2 - 0.45) / 0.36 = -0.69
D_adv = (0.0 - 0.45) / 0.36 = -1.25

This lets GRPO remove the value model / critic:

PPO:  reward function + critic baseline
GRPO: reward function + group-relative baseline

How GRPO Works

Suppose the prompt is a math problem:

If 3x + 5 = 20, what is x?

GRPO does not sample only one answer. It samples a group of answers for the same prompt:

The group mean reward is the baseline, and the group standard deviation normalizes scale:

mean_reward = (1.0 + 0.0 + 1.0 + 0.0) / 4 = 0.5
std_reward  = 0.5

Each answer's advantage can be understood as:

A_adv = (1.0 - 0.5) / 0.5 = +1.0
B_adv = (0.0 - 0.5) / 0.5 = -1.0
C_adv = (1.0 - 0.5) / 0.5 = +1.0
D_adv = (0.0 - 0.5) / 0.5 = -1.0

The update is intuitive:

increase the token probabilities for A / C
decrease the token probabilities for B / D

There is no separate value model. The baseline comes from the group mean for the same prompt; the standard deviation only rescales the advantage. In other words:

PPO asks the critic: roughly how much future reward is this prefix worth?
GRPO asks the group: is this answer better or worse than the other answers for the same problem?

GRPO still usually keeps PPO-like ratio / KL control so the model does not over-optimize reward and drift too far from the reference policy. It is especially suitable for math, code, multiple-choice, and format-verifiable tasks because rewards can be computed automatically:

math: whether the final answer is correct
code: whether unit tests pass
tool use: whether the task is completed and the evidence satisfies the requirement

GRPO is especially natural for RLVR, or reinforcement learning with verifiable rewards:

• Math: the final answer can be checked.
• Code: unit tests can verify behavior.
• STEM / reasoning: a rule-based verifier or judge can be used; if using an LLM-as-judge, watch for judge bias and reward hacking.

GRPO strengths:

• No critic training, so memory and engineering complexity are lower.
• Multiple samples for the same prompt naturally provide relative comparison signals.
• DeepSeekMath introduced GRPO; DeepSeek-R1-Zero used GRPO-style large-scale RL; DeepSeek-R1 then added cold-start SFT, rejection sampling / SFT, and additional RL stages, so R1 should not be simplified to "only GRPO."

GRPO weaknesses:

• It still requires online rollout, so sampling cost is high.
• Rewards should preferably be verifiable; purely subjective preference tasks are not always a good fit.
• Group size, reward normalization, and KL control all affect stability.

12.5 Do Not Confuse Reward Model and Critic

The most interview-relevant distinction:

reward model: answer -> score
critic: partial trajectory/state -> expected return
advantage: per-token/step return estimate (often GAE / KL-shaped) - critic prediction

12.6 How to Choose

The most practical interview answer:

PPO is the traditional RLHF workhorse: complete but heavy. DPO is the preference-alignment workhorse: offline, simple, and stable. GRPO is a reasoning-RL workhorse: it replaces the critic with group-relative reward and is well suited to verifiable math/code-style tasks.

12.7 Common Questions

Is DPO RL?

In a broad sense, it is derived from an RLHF objective. In the narrower, traditional sense, it is not online RL: there is no rollout, no environment interaction, and no critic. It is more accurate to call it offline preference optimization.

Why Does GRPO Not Need a Critic?

Because it samples multiple answers for the same prompt, uses the group reward mean as the baseline, and uses the group standard deviation to normalize advantage. The critic's role is to provide a baseline; GRPO replaces it with a group baseline.

What Is the Biggest Difference Between PPO and GRPO?

PPO usually trains a value model / critic to estimate advantage. GRPO estimates advantage from relative rewards among multiple samples for the same prompt. PPO is more general; GRPO is cheaper and more direct for verifiable reasoning tasks.

What Do All Three Share?

All three control how much the new model's probabilities change relative to an old policy or reference model. PPO uses ratio clipping and KL, DPO uses a reference-relative preference loss, and GRPO usually keeps KL or ratio clipping as well.

12.8 Final Cheat Sheet

PPO:
  online RLHF, reward model/function + critic + KL/clip
  strongest general RLHF baseline, but expensive and sensitive

DPO:
  offline preference optimization, chosen/rejected pair
  no rollout, no explicit reward model, no critic
  simple, stable, good for alignment

GRPO:
  online critic-free RL, group-relative advantage
  no value model, good for verifiable reasoning rewards
  common in modern verifiable reasoning RL

12.9 References

• PPO: Proximal Policy Optimization Algorithms
• InstructGPT / RLHF
• HuggingFace TRL
• DPO: Direct Preference Optimization
• DeepSeekMath / GRPO
• DeepSeek-R1

Apr 2026 snapshot · Classic Hot 100 · PyTorch from scratch

This is a two-hour interview-prep notebook, not a full solution manual. The Hot 100 list here is a Classic Hot 100 snapshot for Apr 2026 interview prep, not a claim that it mirrors LeetCode's live list. The goal is to quickly recover each problem's pattern entry point, the template you should reach for, and the shape reasoning behind three common deep-learning implementation prompts: MHA, Conv1D, and Conv2D.

Core rule: Classic Hot 100 practice is not about memorizing problem numbers. It is about recognizing the state, data structure, or transition as soon as you see the prompt. For implementation questions, the bar is not cleverness; it is clear shape accounting, boundary handling, mask semantics, stride, and padding.

A.1 Two-Hour Review Plan

A.2 Classic Hot 100 Pattern Entry Points

A.3 Classic Hot 100 Solution Table

This table keeps the interview-useful part only: pattern entry point, template, and the one key move. In an interview, state the template first, then handle edge cases.

A.4 Selection Order After Seeing a Prompt

1. Need "contiguous subarray/path sum equals K"? -> prefix sum + hashmap
2. Need "longest/shortest substring"?           -> sliding window
3. Sorted / can sort and find pairs or triples? -> two pointers
4. Linked-list delete/merge/reverse?            -> dummy + pointer rewrite
5. Tree asks path/height/ancestor?              -> DFS return value
6. Matrix connected components?                 -> BFS/DFS + visited
7. Courses/dependencies/order constraints?      -> graph + cycle detection/toposort
8. Optimal value / count / feasibility?         -> DP
9. Enumerate all valid answers?                 -> backtracking
10. Top K / frequency?                          -> heap / bucket / quickselect

A.5 DP and Backtracking Failure Points

For DP, say these four things before coding:

1. state: what dp[i] / dp[i][j] means
2. transition: which previous states it comes from
3. init: empty string, 0, first row / first column
4. answer: return dp[n] or max(dp)

Backtracking template:

def backtrack(path, choices):
    if is_done(path):
        res.append(path.copy())
        return
    for choice in choices:
        if invalid(choice):
            continue
        path.append(choice)
        backtrack(path, next_choices)
        path.pop()

A.6 From-Scratch MHA

Start with shapes. This is usually more important than the exact syntax:

x:      [B, T, C]
q/k/v:  [B, T, C] -> [B, H, T, D]
scores: [B, H, T, T]
out:    [B, H, T, D] -> [B, T, C]

import math
import torch
import torch.nn as nn
import torch.nn.functional as F


class MultiHeadAttention(nn.Module):
    def __init__(self, embed_dim, num_heads, dropout=0.0, bias=True):
        super().__init__()
        assert embed_dim % num_heads == 0
        self.embed_dim = embed_dim
        self.num_heads = num_heads
        self.head_dim = embed_dim // num_heads

        self.q_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.k_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.v_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.dropout = nn.Dropout(dropout)

    def _split_heads(self, x):
        # [B, T, C] -> [B, H, T, D]
        B, T, C = x.shape
        return x.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)

    def _merge_heads(self, x):
        # [B, H, T, D] -> [B, T, C]
        B, H, T, D = x.shape
        return x.transpose(1, 2).contiguous().view(B, T, H * D)

    def forward(self, x, attn_mask=None, key_padding_mask=None, return_attn=False):
        # x: [B, T, C]
        B, T, C = x.shape
        q = self._split_heads(self.q_proj(x))
        k = self._split_heads(self.k_proj(x))
        v = self._split_heads(self.v_proj(x))

        scores = q @ k.transpose(-2, -1)
        scores = scores / math.sqrt(self.head_dim)

        if attn_mask is not None:
            # Attention/causal mask: [T,T] -> [1,1,T,T], or already broadcastable.
            if attn_mask.dim() == 2:
                attn_mask = attn_mask[None, None, :, :]
            elif attn_mask.dim() == 3:
                attn_mask = attn_mask[:, None, :, :]

            # bool True = masked out; binary numeric uses 1=keep, 0=masked.
            # Additive PyTorch masks (0 / -inf) should be added to scores instead.
            if attn_mask.dtype == torch.bool:
                scores = scores.masked_fill(attn_mask, float("-inf"))
            else:
                scores = scores.masked_fill(attn_mask == 0, float("-inf"))

        if key_padding_mask is not None:
            # Padding mask: [B,T]. Do not infer this from a generic 2-D mask.
            key_padding_mask = key_padding_mask[:, None, None, :]
            if key_padding_mask.dtype == torch.bool:
                scores = scores.masked_fill(key_padding_mask, float("-inf"))
            else:
                scores = scores.masked_fill(key_padding_mask == 0, float("-inf"))

        attn = F.softmax(scores, dim=-1)
        attn = self.dropout(attn)
        out = attn @ v
        out = self.out_proj(self._merge_heads(out))
        return (out, attn) if return_attn else out

Common follow-ups:

• Why divide by sqrt(head_dim): it keeps dot-product variance from growing with dimension and prevents softmax saturation.
• Why contiguous(): after transpose, memory is not contiguous; view needs a contiguous layout.
• Where the mask goes: before softmax, fill illegal positions with -inf.
• Causal mask shape: [T, T]; padding mask shape: [B, T]; a general mask should broadcast to [B, H, T, T].

A.7 From-Scratch Conv1D

Input and weight shapes:

x:      [B, Cin, L]
weight: [Cout, Cin, K]
out:    [B, Cout, Lout]
Lout = floor((L + 2P - D*(K-1) - 1) / S + 1)

import torch
import torch.nn.functional as F


def conv1d_naive(x, weight, bias=None, stride=1, padding=0, dilation=1):
    B, Cin, L = x.shape
    Cout, Cin_w, K = weight.shape
    assert Cin == Cin_w

    x = F.pad(x, (padding, padding))
    Lpad = x.shape[-1]
    Lout = (Lpad - dilation * (K - 1) - 1) // stride + 1
    out = x.new_zeros(B, Cout, Lout)

    for b in range(B):
        for co in range(Cout):
            for i in range(Lout):
                start = i * stride
                total = x.new_tensor(0.0)
                for ci in range(Cin):
                    for k in range(K):
                        total = total + x[b, ci, start + k * dilation] * weight[co, ci, k]
                if bias is not None:
                    total = total + bias[co]
                out[b, co, i] = total
    return out

Verification:

torch.manual_seed(0)
x = torch.randn(2, 3, 12)
w = torch.randn(4, 3, 5)
b = torch.randn(4)
y = conv1d_naive(x, w, b, stride=2, padding=2)
ref = F.conv1d(x, w, b, stride=2, padding=2)
assert torch.allclose(y, ref, atol=1e-5)

A.8 From-Scratch Conv2D

Input and weight shapes:

x:      [B, Cin, H, W]
weight: [Cout, Cin, KH, KW]
out:    [B, Cout, Hout, Wout]
Hout = floor((H + 2PH - DH*(KH-1) - 1) / SH + 1)
Wout = floor((W + 2PW - DW*(KW-1) - 1) / SW + 1)

import torch
import torch.nn.functional as F


def _pair(v):
    return v if isinstance(v, tuple) else (v, v)


def conv2d_naive(x, weight, bias=None, stride=1, padding=0, dilation=1):
    stride_h, stride_w = _pair(stride)
    pad_h, pad_w = _pair(padding)
    dil_h, dil_w = _pair(dilation)

    B, Cin, H, W = x.shape
    Cout, Cin_w, KH, KW = weight.shape
    assert Cin == Cin_w

    x = F.pad(x, (pad_w, pad_w, pad_h, pad_h))
    Hpad, Wpad = x.shape[-2], x.shape[-1]
    Hout = (Hpad - dil_h * (KH - 1) - 1) // stride_h + 1
    Wout = (Wpad - dil_w * (KW - 1) - 1) // stride_w + 1
    out = x.new_zeros(B, Cout, Hout, Wout)

    for b in range(B):
        for co in range(Cout):
            for oh in range(Hout):
                for ow in range(Wout):
                    h0 = oh * stride_h
                    w0 = ow * stride_w
                    total = x.new_tensor(0.0)
                    for ci in range(Cin):
                        for kh in range(KH):
                            for kw in range(KW):
                                total = total + (
                                    x[b, ci, h0 + kh * dil_h, w0 + kw * dil_w]
                                    * weight[co, ci, kh, kw]
                                )
                    if bias is not None:
                        total = total + bias[co]
                    out[b, co, oh, ow] = total
    return out

Verification:

torch.manual_seed(0)
x = torch.randn(2, 3, 8, 9)
w = torch.randn(4, 3, 3, 5)
b = torch.randn(4)
y = conv2d_naive(x, w, b, stride=(2, 1), padding=(1, 2))
ref = F.conv2d(x, w, b, stride=(2, 1), padding=(1, 2))
assert torch.allclose(y, ref, atol=1e-5)

A.9 How to Talk Through the Three Implementation Questions

A.10 Last Five-Minute Notes

Hot 100:
  sum/count -> hashmap/prefix
  substring -> sliding window
  sorted pair/triple -> two pointers
  tree -> define DFS return
  graph/matrix -> visited + BFS/DFS
  optimal value -> DP state first

MHA:
  [B,T,C] -> [B,H,T,D] -> scores [B,H,T,T] -> softmax -> merge

Conv:
  output size = floor((input + 2*pad - dilation*(kernel-1) - 1)/stride + 1)

A.11 References

• LeetCode Top 100 Liked study plan: https://leetcode.com/studyplan/top-100-liked/
• PyTorch nn.MultiheadAttention: docs
• PyTorch nn.Conv1d: docs
• PyTorch nn.Conv2d: docs