Long-Horizon Agents: State, Planning, and Verifiable Hours-Long Execution

A long-horizon agent is not one whose timeout moved from ten minutes to ten hours, nor one that pours its entire history into a larger context. It is a recoverable state machine: it maintains goals and constraints in a changing environment, decomposes plans into verifiable local actions, resumes from the right boundary after failure, and lets an external evaluator determine whether the final result is task completion, a reward proxy, or merely a trajectory that looked busy.

1. What Counts as Long Horizon, and What Does Not

A long-running episode has at least four properties: state remains causally relevant across decisions; intermediate errors change later feasible paths; the agent trades local feedback against global constraints; and after interruption it must reconstruct from external artifacts rather than guess from chat memory. Under this definition, the difficulty is not merely “more steps,” but dependencies, recovery, and accountability.

WildClawBench is valuable because it restores the native CLI harness to the evaluation configuration: in its containerized tasks, swapping harnesses for the same model can substantially change performance. It supports attributing long-horizon performance to a model-harness configuration, not assigning a runtime-free fixed ranking to a model.

2. Label the Task Envelope First: Long Tasks, Single-Project Challenges, and Scientific Workflows Are Different Claims

“Long horizon” describes execution and dependency, not one uniform task type. LongCLI-Bench separates requirement fulfilment from regression avoidance with a dual-set measurement, showing that one terminal test does not localize long-horizon programming failure. Terminal-Bench Challenges instead are token-intensive, single-project tasks. Their run guidance explicitly permits custom stateful rollouts to manage checkpoints, orchestration, retries, and memory while retaining task source and verifier as the contract.

By contrast, as of 2026-07-13, Terminal-Bench Science is still in development. It targets computational workflows in the natural sciences with containerized, programmatic, deterministic verification and explicitly excludes open-ended hypothesis generation and literature review. It can support study of whether an agent completes a constrained, reviewable scientific-computing workflow; it is not a released science-benchmark score, evidence of an independent discovery, or evidence of a days-long service.

Every report should therefore name its task envelope before saying how long the agent ran. Otherwise “science,” “challenge,” and “long horizon” collapse fundamentally different verifiers, clocks, and conclusions into one score.

3. Memory and State: From Similarity Retrieval to Recoverable, Verifiable Execution State

Whatever the task envelope, state in a long-horizon agent is not a chat summary. Agent memory is often reduced to “put history into a vector store and retrieve it when needed.” That can help question answering or personalization, but it is not enough for long-horizon action. An agent also needs to know what was committed, what is only planned, which observations are stale, which failed branches must not execute again, and which memories must not affect authority or final judgment. The research object is an execution-state system with provenance, temporal validity, visibility, authority, and recovery semantics.

3.1 Separate Four Different Kinds of Memory

This explains why a longer context is not a memory system: it declares neither provenance nor whether a write committed, and it cannot resolve conflict, staleness, or recovery. For an agent, context is input to one computation; memory is external state and an access protocol across steps, sessions, and potentially agents.

3.2 From Stored Text to Managed Memory Operations

Real agent memory arises from continuous agent-environment interactions rather than only human dialogue. AMA-Bench observes that similarity retrieval without causal and objective information limits performance. Agentic Memory / AgeMem exposes long- and short-term memory management as tool actions the agent can call, including store, retrieve, update, summarize, and discard. It trains and evaluates this policy in its own five long-horizon benchmarks. That supports a concrete idea: memory operations can be part of a policy rather than fixed heuristics; it does not show that every RL memory controller generalizes or is safe.

Agent Memory separates construction, retrieval, and generation phases from a systems perspective, characterizing how designs move cost onto write or read paths and discussing freshness-latency trade-offs. It is a reminder that a reported “memory improvement” may be more frequent extraction, a larger context, or more generation calls, all of which need separate accounting.

MAGE organizes long-horizon memory as a hierarchical execution-state tree. Its active root-to-current path combines current subgoals, recent traces, and validated hints; Grow, Compress, Maintain, and Revise respectively write, compress, validate, and restore onto a branch. Its results hold only in the paper’s MemoryArena setting, but it provides a sharper object than “top similar passage”: measure state lineage, branch isolation, and error recovery.

3.3 Memory Is Execution State, Not “Smarter Retrieval”

For tasks with external side effects, the most important state is often not a knowledge passage but a constraint on action:

MemoryEvent     = source, observed_at, visible_to, trust_label, digest, temporal_validity
ExecutionState  = goal, constraints, committed_effects, pending_effects, artifact/env_digest
Checkpoint      = state_version, authority_snapshot, plan, budget_clock, recovery_policy
RetrievalRecord = query, candidate_ids, admission, excluded_items, reason, cost
Acceptance      = independent_postcondition, evidence_digest, verifier_version, outcome

This creates five key distinctions:

1. Facts, inferences, and plans differ. “The user asked to pay,” “the balance is 100,” “prepare payment,” and “payment committed” are different types, not one summary sentence.
2. Visible differs from true. Screenshots, DOM, tool responses, and chat are observations; backend state, file digests, and independent verifiers offer stronger postcondition evidence.
3. Failed branches differ from the active path. A known failed attempt can aid diagnosis but must not silently enter active state and induce repeated side effects.
4. Retrieval differs from authority. An old preference, web hint, or tool description may prioritize retrieval; it cannot grant write access, widen a domain, or modify a verifier.
5. Memory differs from acceptance. Memory can help choose action; final completion still belongs to a postcondition verifier the agent cannot write.

This aligns with the world-model and environment contract in Agent Research Environments: state transition, observation provenance, action authority, reset/recovery, and verifier are different objects, and memory cannot substitute for any of them. Every cross-context handoff should therefore reconstruct at least the following from external state rather than a prior-turn summary.

Context compaction is not a checkpoint. The former may help a fresh invocation understand history; the latter must also locate environment, artifacts, and authority. If an external service cannot return to its prior state, retain it as observational evidence rather than pretending it can be rerun deterministically.

3.4 What Current Benchmarks Measure, and What They Do Not

The narrow conclusion common to these sources is that memory needs its own measurement and that tasks expose different failure modes. They do not provide a universal memory winner or permit chat-QA accuracy to be extrapolated to reliable web, terminal, or research workflows with real side effects.

3.5 A Memory Read Is a Trust Boundary

Beyond Similarity studies memory search in personal agents and treats semantically related but contextually inappropriate retrieval as a control-plane risk: it can lead to cross-domain leakage, sycophancy, tool-call drift, or memory-induced jailbreaks. The paper evaluates specified frameworks, environments, and attacks, so it is not a prevalence estimate for all memory products. It does support one necessary design rule: retrieved does not mean permitted to affect action.

Memory admission should therefore check all of these surfaces:

Treating memory as untrusted input aligns with the tool-governance principle in Web-Agent Environments: tool returns, pages, references, scripts, and memory items can affect policy, while only runtime capability and an independent verifier control real side effects.

3.6 State Recovery Is Not Learning from Experience

Recovery answers whether the same work can continue after interruption; continual learning answers whether earlier task experience makes later tasks better under the same capability and budget conditions. Continual Learning Bench (CL-Bench) designs task sequences around shared latent structures that a stateful system can discover online and uses a gain metric to separate prior capability from experience-dependent improvement. Its results in a specific six-domain setting remind us that a dedicated memory system does not automatically yield reliable online learning; putting history into context, a store, or a skill library alone does not show that a system generalizes from experience.

The distinction below is a reporting protocol proposed here, not a universal experimental finding from any one paper:

Any system said to “get better with use” should also record where experience enters: context, an event ledger, retrieval memory, a skill, policy/adapter parameters, or a harness/scheduler. Fix the base model, harness, tool authority, task order, total token/wall-clock budget, and an independent final sequence; compare no experience, checkpoint/retrieval only, and the declared learning update. Report initial_capability, sequential_gain, forgetting/regression, update cost, leakage, and final verified outcome separately. Otherwise a stronger starting model, extra context, more attempts, or information from future tasks can masquerade as learning.

3.7 Trainable Memory Is Not a Training Set of “Successful Summaries”

AgeMem shows that store/retrieve/update can be learnable actions; MemGym shows that memory’s effect can be separated, as far as the benchmark permits, from reasoning, retrieval, and tools. This gives synthetic instances and agentic RL a rigorous entry point, but it also raises a stronger isolation requirement: during training, memory cannot reveal final tasks, hidden verifiers, successful evaluation traces, or an evaluator’s answer channel.

A falsifiable training protocol fixes base model, harness, tool policy, action authority, total token/rollout/wall-clock budget, and hidden verifier, then compares:

Final evaluation should be separated at least by environment, workflow/rule, memory schema or generator family, and privacy/security family. When a generator or training trace has seen a test’s unique state template, hidden postcondition, or successful action, a memory gain is simply another form of test leakage. See the synthetic-training sections of Agent Research Environments for the data-lineage and split protocol.

3.8 A Memory Failure Suite Sharp Enough to Matter

Do not only ask “did it remember?” Every memory system should confront at least these controls:

1. Stale override: Later high-trust state conflicts with an earlier low-trust record; the agent must update or abstain.
2. Partial commit: A tool return says success while the backend commits only part; recovery must not repeat the external side effect.
3. Failed-branch contamination: A disproven plan is semantically similar to a valid trace and must not return as current fact.
4. Untrusted write: Web/tool text induces storage, summary, or forwarding of an unauthorized instruction.
5. Cross-scope leakage: Different user/task/tenant memories in one agent cannot influence one another’s retrieval or actions.
6. Irreversible compression: A summary loses an artifact digest, constraint, or commit status such that an independent runner cannot reconstruct key state.
7. Recovery and stopping: With missing checkpoints, tool denial, or environment drift, the agent recovers, escalates, or safely stops rather than fabricating completion.

The minimum output columns are verified_success, state_reconstruction, stale_or_conflict_error, memory_admission_violation, cross_scope_leak, duplicate_side_effect, recovery_cost, safe_stop, and trace_completeness. They make memory an object that environment, harness, and evaluator can audit together rather than a vague capability label.

3.9 What Memory Adds to the Five Core Threads

In short, agent memory is not extra “experience” placed into a prompt but a state system that changes future action — its value lies not in retrieving more, but in making each write, read, compression, recovery, and revocation definable, constrained, auditable, and independently falsifiable. It is the first layer a long-horizon agent must keep under control; the next is planning.

4. Planning and Replanning: From Plausible Plans to Verifiable Replanning

The next layer above execution state is the plan. A clear, orderly plan in text does not mean an agent can plan. A real plan must keep updating under partial observations, hard constraints, tool failures, authority limits, resource budgets, and environment changes, then be accepted by an independent terminal state rather than the agent’s explanation. The research object is not “a plan that sounds reasonable,” but a control process that an environment can refute, that can replan safely, and that correctly stops when work is infeasible.

4.1 Separate Five Objects First

Agent Planning Benchmark (APB) starts from this gap. It separates holistic planning, feedback-conditioned step-wise planning, distracting/broken tools, and unsolvable tasks as upstream diagnostics for end-to-end execution benchmarks. Its evidence is scoped to its tasks and models, but it supports a necessary rule: do not assign final failure automatically to planning, or a good-looking plan automatically to execution ability.

4.2 A Contract for an Executable Plan

A natural-language plan in chat drops boundaries needed for execution, comparison, and recovery. A minimal plan version should retain:

GoalSpec       = target_postcondition, hard_constraints, soft_preferences, budget, authority
PlanVersion    = parent, state_digest, assumptions, subgoals, dependencies, stop_conditions
PlanStep       = precondition, authorized_action, expected_observation, expected_cost, recovery_boundary
Observation    = source, timestamp, provenance, state_delta | uncertainty | tool_failure
ReplanEvent    = trigger, invalidated_steps, alternatives, chosen_action, budget_delta, rationale
Acceptance     = independent_postcondition, evidence_digest, verifier_version, outcome

This does not require every agent to emit verbose JSON. It requires the harness to preserve equivalent information externally. It creates six auditable questions:

1. Can the goal be acceptance-checked? A goal points to a checkable postcondition, not merely “finish the research” or “handle the request.”
2. Are constraints explicit? Time, cost, authority, irreversible actions, privacy, and policy are hard or soft constraints, not prose that can disappear.
3. Why is each step allowed? Preconditions, authority, and expected observations prevent a vaguely related tool call from concealing an unauthorized or ineffective action.
4. When should replanning occur? Failure, absent signal, state drift, concurrent conflict, budget change, and uncertainty are explicit triggers.
5. What does replanning affect? Record invalidated steps to avoid repeating committed side effects or treating a failed branch as current fact.
6. When should the system refuse or stop? Infeasibility, missing critical information, insufficient authority, or exhausted budget make refusal/safe stop correct outcomes, not low-scoring trajectories.

4.3 A Plan Must Face Global Constraints, Hidden Tools, and Real Failure

DeepPlanning places active information gathering, local constraints, and global time/budget optimization in one long-horizon task. It reminds us that locally correct choices do not make a globally feasible plan, and a sequence of locally plausible actions can still violate overall feasibility, particularly when tools fail, information is absent, or a route must change.

PlanBench-XL combines a large tool ecosystem, limited tool visibility, implicit subgoals, and optional missing, broken, or distracting tools. It distinguishes planning over a known fixed tool list from planning that must discover tools and use returned evidence to choose the next step. Failures without explicit error signals especially cannot be judged by one successful tool-call response.

This is why browser, terminal, scientific, and cross-application API tasks need different verifiers. In AutomationBench, a cross-application REST workflow additionally requires endpoint discovery, layered business policy, and correct data writes; the paper grades programmatic end states. It is a useful example of backend terminal state constraining planning, not a generalization to every UI, browser environment, or business process.

The design lesson from VLAs-as-Tools also applies to software agents: a high-level agent keeps temporal plans and recovery, bounded local tools perform small tasks, and replanning is triggered by explicit progress or exception events rather than constant polling. BCER Agent binds final outputs to intermediate artifacts and measurements and emphasizes bounded local recovery. These works are not evidence of a universal winning architecture; they expose two testable variables: whether progress feedback is verifiable and whether recovery rolls back only to the affected local boundary. The plan-execute loop can therefore be inspected phase by phase:

4.4 Replanning Is a Controlled Change, Not Infinite Retry

High-quality replanning does not turn every failure into “try again.” Each ReplanEvent should name the triggering observation, invalidated assumptions or steps, affected artifact/state, any irreversible effect, remaining budget, and how the replacement plan will be independently checked.

Making these explicit actions lets us measure recovery quality: replan_trigger_precision, invalidated_step_accuracy, duplicate_side_effect, recovery_cost, safe_stop, and verified_success_after_replan. For long horizons, replanning connects to the checkpoint/branch contract in the memory section above: plan versions and environment state must recover together; a chat summary is not enough.

4.5 Multi-Agent Systems, Harness Edits, and GPU Scheduling Are Not Planning

ClawArena-Team evaluates a leader managing a fixed subagent pool and jointly scores task correctness with least privilege and modality routing. It separates team management from workers’ raw task solving, showing why delegation plans need separate authority, cost, and final-outcome measures.

Agentic Harness Engineering (AHE) makes editable harness components, trajectory experience, and each edit’s prediction/later verification observable. In its coding-agent setting, this edit-to-outcome connection is a controlled form of harness evolution. It does not establish that arbitrary self-editing harnesses or workflow search are RSI; evaluator, authority, and held-out acceptance must remain external to the candidate.

SAGA tackles a different layer: treating complete agent workflows as GPU scheduling units to improve task-completion latency. It is a resource-scheduling contribution for deployment, not evidence of policy planning. A system that executes the same workflow faster still needs a separate check that the workflow picked the right tools, respected constraints, and reached the terminal state.

4.6 An Evaluation Matrix That Can Falsify “Can Plan”

Under the same base model, tool policy, action authority, environment initial state, total token/wall-clock/worker budget, and hidden verifier, compare at least:

The minimum counterexample set contains correct endpoint plus wrong payload; successful response with no commit; missing or silently failing tools; stale state; unsolvable tasks; budget conflicts; late subagent returns; and trajectories that improve a process score without passing hidden postconditions. A rising plan score must then meet execution, integrity, and terminal-state evidence together.

4.7 Synthetic Instances and Agentic RL: Train Replanning Without Leaking Evaluation

A synthetic instance cannot supply only one successful plan. A training-useful planning instance has initial-state and visibility boundaries, goal and hard constraints, candidate tools and authority, executable transitions, perturbation/failure branches, alternate paths, refusal conditions, and an independent verifier. Training traces may show when a plan is proposed, revised, or abandoned; they cannot contain the final evaluator, hidden postcondition, successful evaluation trace, or unique answer template.

Compare SFT-only, RL-only, and SFT+RL on the same action surface, and separate final tasks by environment, workflow/rule, tool-schema/generator family, and failure/security family. Reward must report verified_outcome, hard-constraint violations, integrity violations, recovery cost, and correct refusal separately. One scalar writable by the planner or agent cannot decide every conclusion. See the synthetic-training sections of Agent Research Environments and the Agentic RL section of Agent Harness Engineering.

4.8 What Planning Adds to the Five Core Threads

The conclusion here is equally modest: a plan is an inspectable intermediate artifact, not a capability declaration. Only when it updates, refuses, or completes safely under changing state, limited tools, real failures, global constraints, and an independent verifier does “the agent can plan” become an engineering conclusion.

5. Measuring Recovery: Fault Injection and the Recovery Ledger

Section 4 treated recovery actions as part of replanning; this section measures recovery on its own terms. Good recovery has a boundary: it must not rerun a full workflow to improve pass rate while hiding failure sources. An auditable recovery event records its trigger, affected scope, recovery point, invalidated evidence, retry budget, human handoff, and terminal outcome.

A minimal fault-injection set can include incomplete tool results, externally modified file or service state, corrupted checkpoints, denied authority, a summary that omits a key constraint, and a local success that already violates the global budget. For each fault, report recovery_success, state_reconstruction, duplicate_side_effect, resume_latency, and recovery_cost. If recovery raises a process score without improving independent terminal states or safety, it is another gameable reward proxy.

6. Long-Horizon Security Is Not a Repeated Single-Turn Guard

AgentLAB places intent hijacking, tool chaining, task injection, objective drift, and memory poisoning in multi-turn environments. Its narrow conclusion is that single-turn defenses do not automatically cover multi-turn attacks. For long-running agents, the state store, tool outputs, handoff documents, and subagent messages are all untrusted-input surfaces; a prior approval does not automatically authorize the next turn. This is the same trust boundary as the memory admission above: neither storage nor retrieval automatically confers authority.

Security state therefore belongs in the checkpoint: time-bounded authority, denied actions, domain/tool allowlists, trust labels on external inputs, risk escalation, and human confirmations. Final reports should at minimum separate task_outcome, integrity_violations, unsafe_action_rate, and false_block_rate; one success rate cannot cover all four.

7. “Hours” Require an Auditable Time Ledger

A wall-clock number alone is not explanatory. Each run should split wall_clock into active_agent_time, environment_build_or_reset, model_or_tool_wait, external_or_queue_wait, recovery_time, human_handoff_time, and cancelled_or_idle_time, then bind them to the same run id as tokens, tool calls, and external cost. That makes it possible to distinguish sustained agent work from system waiting, state reconstruction, or human takeover.

A fair comparison freezes the cutoff and continuation policy: when to checkpoint, when recovery is allowed, when human escalation is required, and when to safe-stop. One run can be long mostly because it waits in a queue; another can be short yet reconstruct state under a fault. They are different evidence.

8. A Falsifiable Protocol for Hours-Long Execution

The following is a design proposal, not a reported long-horizon result. It makes “works continuously for hours” a claim that can fail:

1. Freeze task envelope and run identity. Retain task type, task manifest, initial-state digest, model/harness/tool/policy versions, budget, environment image, and evaluator hash; do not describe an in-progress benchmark’s target task count as a released result.
2. Externalize state. Each checkpoint writes goals, constraints, artifacts, dependencies, actions, observations, failures, and resources; redact sensitive content before retention.
3. Test recovery actively. Preregister fault-injection points and success definitions; distinguish observational replay from genuine reruns.
4. Separate acceptance. A candidate cannot write hidden tasks, verifier, authority policy, or final release decision; judge final state and process integrity separately.
5. Compare at equal budget. Single-agent, sequential, multi-agent, memory, and recovery variants receive the same total tokens, worker time, wall clock, retries, and external-tool allowance.
6. Report the complete curve and time ledger. Report attempts, cancellations, recovery, regression, variance, peak concurrency, active/wait/recovery/human time, total cost, and cost per verified success, not only the best run.

9. How It Returns to the Core Five

Sections 3.9 and 4.8 listed what memory and planning add to the five core threads; the table below maps the long-horizon thread onto the adjacent threads.

This protocol complements Agent Harness Engineering: that note defines runtime engineering planes; this one turns them into task-envelope-specific long-horizon execution, fault, and acceptance experiments, and unpacks memory/execution state and planning/replanning as two of the controlled layers within them. See AutoResearch for when a scientific workflow can become a reviewable research claim and for adjacent memory, planning, and safety directions; Agent Research Environments for synthetic training, environment boundaries, and the state-transition and verifier contract; and Web-Agent Environments for tool governance.