Agent Memory: From Similarity Retrieval to Recoverable, Verifiable Execution State

Agent memory is often reduced to “put history into a vector store and retrieve it when needed.” That can help question answering or personalization, but it is not enough for long-horizon action. An agent also needs to know what was committed, what is only planned, which observations are stale, which failed branches must not execute again, and which memories must not affect authority or final judgment. The research object is an execution-state system with provenance, temporal validity, visibility, authority, and recovery semantics.

1. Separate Four Different Kinds of Memory

This explains why a longer context is not a memory system: it declares neither provenance nor whether a write committed, and it cannot resolve conflict, staleness, or recovery. For an agent, context is input to one computation; memory is external state and an access protocol across steps, sessions, and potentially agents.

2. From Stored Text to Managed Memory Operations

Agentic Memory / AgeMem exposes long- and short-term memory management as tool actions the agent can call, including store, retrieve, update, summarize, and discard. It trains and evaluates this policy in its own five long-horizon benchmarks. That supports a concrete idea: memory operations can be part of a policy rather than fixed heuristics; it does not show that every RL memory controller generalizes or is safe.

Agent Memory separates construction, retrieval, and generation phases from a systems perspective, characterizing how designs move cost onto write or read paths and discussing freshness-latency trade-offs. It is a reminder that a reported “memory improvement” may be more frequent extraction, a larger context, or more generation calls, all of which need separate accounting.

MAGE organizes long-horizon memory as a hierarchical execution-state tree. Its active root-to-current path combines current subgoals, recent traces, and validated hints; Grow, Compress, Maintain, and Revise respectively write, compress, validate, and restore onto a branch. Its results hold only in the paper’s MemoryArena setting, but it provides a sharper object than “top similar passage”: measure state lineage, branch isolation, and error recovery.

3. Memory Is Execution State, Not “Smarter Retrieval”

For tasks with external side effects, the most important state is often not a knowledge passage but a constraint on action:

MemoryEvent     = source, observed_at, visible_to, trust_label, digest, temporal_validity
ExecutionState  = goal, constraints, committed_effects, pending_effects, artifact/env_digest
Checkpoint      = state_version, authority_snapshot, plan, budget_clock, recovery_policy
RetrievalRecord = query, candidate_ids, admission, excluded_items, reason, cost
Acceptance      = independent_postcondition, evidence_digest, verifier_version, outcome

This creates five key distinctions:

1. Facts, inferences, and plans differ. “The user asked to pay,” “the balance is 100,” “prepare payment,” and “payment committed” are different types, not one summary sentence.
2. Visible differs from true. Screenshots, DOM, tool responses, and chat are observations; backend state, file digests, and independent verifiers offer stronger postcondition evidence.
3. Failed branches differ from the active path. A known failed attempt can aid diagnosis but must not silently enter active state and induce repeated side effects.
4. Retrieval differs from authority. An old preference, web hint, or tool description may prioritize retrieval; it cannot grant write access, widen a domain, or modify a verifier.
5. Memory differs from acceptance. Memory can help choose action; final completion still belongs to a postcondition checker the agent cannot write.

This aligns with the contract in World Models and Agent Environments: state transition, observation provenance, action authority, reset/recovery, and verifier are different objects. Memory cannot substitute for any of them.

4. What Current Benchmarks Measure, and What They Do Not

The narrow conclusion common to these sources is that memory needs its own measurement and that tasks expose different failure modes. They do not provide a universal memory winner or permit chat-QA accuracy to be extrapolated to reliable web, terminal, or research workflows with real side effects.

5. A Memory Read Is a Trust Boundary

Beyond Similarity studies memory search in personal agents and treats semantically related but contextually inappropriate retrieval as a control-plane risk: it can lead to cross-domain leakage, sycophancy, tool-call drift, or memory-induced jailbreaks. The paper evaluates specified frameworks, environments, and attacks, so it is not a prevalence estimate for all memory products. It does support one necessary design rule: retrieved does not mean permitted to affect action.

Memory admission should therefore check all of these surfaces:

Treating memory as untrusted input aligns with Agent Tool Governance: tool returns, pages, references, scripts, and memory items can affect policy, while only runtime capability and an independent verifier control real side effects.

6. Trainable Memory Is Not a Training Set of “Successful Summaries”

AgeMem shows that store/retrieve/update can be learnable actions; MemGym shows that memory’s effect can be separated, as far as the benchmark permits, from reasoning, retrieval, and tools. This gives synthetic instances and agentic RL a rigorous entry point, but it also raises a stronger isolation requirement: during training, memory cannot reveal final tasks, hidden verifiers, successful evaluation traces, or an evaluator’s answer channel.

A falsifiable training protocol fixes base model, harness, tool policy, action authority, total token/rollout/wall-clock budget, and hidden evaluator, then compares:

Final evaluation should be separated at least by environment, workflow/rule, memory schema or generator family, and privacy/security family. When a generator or training trace has seen a test’s unique state template, hidden postcondition, or successful action, a memory gain is simply another form of test leakage. See Synthetic Instances for Agent Training for the data-lineage and split protocol.

7. A Failure Suite Sharp Enough to Matter

Do not only ask “did it remember?” Every memory system should confront at least these controls:

1. Stale override: Later high-trust state conflicts with an earlier low-trust record; the agent must update or abstain.
2. Partial commit: A tool return says success while the backend commits only part; recovery must not repeat the external side effect.
3. Failed-branch contamination: A disproven plan is semantically similar to a valid trace and must not return as current fact.
4. Untrusted write: Web/tool text induces storage, summary, or forwarding of an unauthorized instruction.
5. Cross-scope leakage: Different user/task/tenant memories in one agent cannot influence one another’s retrieval or actions.
6. Irreversible compression: A summary loses an artifact digest, constraint, or commit status such that an independent runner cannot reconstruct key state.
7. Recovery and stopping: With missing checkpoints, tool denial, or environment drift, the agent recovers, escalates, or safely stops rather than fabricating completion.

The minimum output columns are verified_success, state_reconstruction, stale_or_conflict_error, memory_admission_violation, cross_scope_leak, duplicate_side_effect, recovery_cost, safe_stop, and trace_completeness. They make memory an object that environment, harness, and evaluator can audit together rather than a vague capability label.

8. Return to the Five Core Threads

The conclusion is direct: agent memory is not extra “experience” placed into a prompt. It is a state system that changes future action. Its research value lies not in retrieving more, but in making each write, read, compression, recovery, and revocation definable, constrained, auditable, and independently falsifiable. See Long-Horizon Agents for the broader checkpoint and fault protocol, and Agent Planning for a separate contract for plan versions, feedback-conditioned replanning, refusal, and terminal acceptance; memory is one controlled layer within it, never its replacement.