AI Agent Memory Benchmarks: What Two Small Studies Show Us

Our early AI agent memory work explored two different questions: whether selective retrieval can provide the right context, and whether a predefined workflow can reduce repeated orchestration. This article explains the study designs and the measurement lessons. The original numerical results are not republished here because the underlying run records were not available for verification during this review.

For a team choosing what to test next, the useful distinction is between selecting relevant memory and orchestrating an entire workflow. The first study measured retrieval and answers under seeded conditions. The second compared two complete workflow designs. Neither measured the everyday benefit of an installed plugin.

A note on small samples, explained simply

These were small studies using authored tasks. Repeating similar tasks can produce reassuring-looking results without showing how a system behaves on unfamiliar work. Treat a component test as evidence about its defined setup, and require a separate installed-workflow evaluation before making a broader claim.

Why we tightened the controls first

The original study account recorded a grading correction after a structurally valid result misread task semantics. That history illustrates why schema checks are not enough. A reviewer should independently read the question, expected evidence and answer, then keep rejected and corrected judgments distinguishable.

Context Manager v3: what we tested

The memory design compared a direct-memory baseline with a selective candidate that could use a bounded lexical fallback for weak catalog matches. A separate empty-memory condition tested whether the authored question actually required retrieved context. That diagnostic condition belongs beside the comparison, not inside the treatment effect.

A reviewable protocol separates qualification from the main cohort. It defines retry and replacement rules before execution, records failed attempts and identifies any deviation. Qualification must not quietly become an opportunity to tune against the held-out examples.

Measure resolution separately from retrieval quality

A resolved task does not tell you whether retrieval was efficient or whether the answer used the correct evidence. The empty-memory condition helps identify questions that can be answered without memory. Keep those cases visible when interpreting retrieval behavior.

Check useful context and unwanted context together

Selecting fewer sources is useful only if the necessary context remains available and the final answer is supported. Measure required-source recall and irrelevant or forbidden-source exposure together. A smaller bundle that still includes distracting material is not clean isolation.

A new test set that no longer reproduces an old failure cannot establish a repair by itself. To compare before and after, keep the relevant case, acceptance rule and conditions stable, then separately test whether the result generalizes to new cases.

Provider usage stayed host-specific

Keep provider usage categories separate and retain the definitions supplied by each host. Ordinary input, cache creation, cached reads and elapsed time may describe different work. Do not rank providers by adding unlike fields or replacing missing telemetry with zero.

Local retrieval work also has a cost: scanning records, reading files and applying fallback logic. Record that work separately from provider tokens. Preserve both the requested model and the model identity actually reported; an unavailable served-model field limits what can be concluded.

Safe answers and resolved tasks are not the same thing

Score task resolution and answer acceptability independently. An answer can reach a correct conclusion while citing an unrelated record. An abstention can be appropriate for safety yet leave a task unresolved. The rubric should state which evidence and behavior each judgment requires.

A narrow sample may hit a ceiling where every tested outcome looks the same. That is a reason to add harder independent cases, not to claim equivalence. Statistical summaries need the sampling assumptions and failure cases that make them interpretable.

A grading correction we want on the record

The grading history is a reminder to test the evaluator. Include a memory-dependent question with no supporting memory, a self-contained question, an unsupported citation and an appropriate abstention. Check that the rubric distinguishes them before scoring the cohort.

A seeded-memory study tests retrieval and answering with prepared records. It does not establish automatic capture, cross-session durability, activation or unrestricted installed-plugin behavior. Those are separate questions that need their own runs and acceptance criteria.

Procedural Memory v2: comparing two full workflow designs

The workflow design contrasted a model-directed controller with a fixed content-generation graph. One design selected the next stage during execution; the other followed predefined transitions and a rubric gate. Comparing them concerns two complete designs, not a single isolated prompt change.

A paired comparison needs outcomes for both members of each pair. If one run times out, retain it in the completion record and identify which pairs remain available for a complete-case comparison. Do not silently replace the failure or describe the surviving pairs as the whole plan.

Timeouts can leave response usage, model identity or request identifiers unavailable. Record that absence explicitly. A missing response does not establish zero token use or prove that the provider did no work.

Report completion before performance

Separate the planned cohort, all attempted runs and completed matched pairs. A useful report lets the reader see which denominator applies to every statement and which failures were excluded from a descriptive timing comparison.

Report pair-level variation alongside any aggregate. A pooled token or timing difference can hide opposite directions between tasks. Without verified raw records, this article makes no claim about the size or direction of a performance difference.

To attribute an effect to a single component, hold the surrounding workflow stable. If orchestration, gates and prompts all differ, describe the comparison as a whole-workflow evaluation. That may still be useful, but it answers a broader question.

Make the quality check hard enough to matter

A quality rubric should test the work that makes an output useful: factual support, required links, source revisions, scope and prohibited actions. A link check on a task requiring no links cannot demonstrate citation handling. Masking labels also requires checking file paths and metadata for clues.

Devflow and Harbor: readiness, not results

The original account also discussed Devflow and a planned Harbor study. Source tests and review can establish readiness to attempt a study; they do not establish an installed-runtime result. This article makes no current readiness, run-count or outcome claim for that separate work.

What these AI agent memory benchmarks do not show

This review does not establish productivity gain, quality improvement, reliability improvement, cost savings, provider superiority or statistical equivalence. The retained material explains methods and recorded evaluation problems. Verified run artifacts would be needed before restoring quantitative findings.

For more detail on how we evaluate installed plugin behavior in practice, see our Claude Code plugin evals field guide. Our broader resource library is available here.

Next tests

A useful next evaluation would use conflicting source revisions, facts distributed across documents, distractor claims and required citations. Test explicit local-only or no-publish instructions as acceptance criteria. For memory, include a real capture session followed by a separate retrieval session with unrelated records. Freeze the setup, verify the control actually omits the treatment and retain the complete attempt record.