What We Learned Evaluating Agent Memory:The Results (Part 2)

This is the second part of a two-part evaluation of Couchbase Agent Memory. If you haven’t read Part 1, the benchmarks, metrics, and pipeline architecture are all explained there. This part describes what happened when we actually ran the experiments. Note: Not all of it went the way we expected.

What we tried and what happened

Memory format: raw messages vs. summaries

The first variable we tested was whether to store raw conversation turns or LLM-generated summaries.

The intuition for summaries is reasonable: conversations are noisy, and a clean summary might give the retriever a better signal. Summaries also reduce token cost significantly, from approximately 13,000 tokens down to 6,200 for the same k on LME-S.

The result: raw messages outperform summaries overall on all three datasets. On LME-S, raw messages score 0.637 vs. 0.614 for summaries. On LME-M, the gap is wider: 0.724 vs. 0.574. On LoCoMo, 0.826 vs. 0.675.

The explanation is the SSA category. Single-Session Assistant questions ask about something the assistant said in a prior session, and they often require exact wording. Summarization discards proper nouns, specific recommendations, and precise phrasing. SSA drops from 0.982 to 0.768 on LME-S when switching from raw messages to summaries. On LME-M, it drops from 0.946 to 0.714. On LoCoMo, something specific like “a tech for good convention” becomes “a convention” after summarization, which breaks any reasoning chain that depends on matching that entity.

Summaries are not without merit. They improve SSP and MS scores because compressing information across turns reduces noise and makes cross-session connections easier. For preference questions, compression tends to surface preference signals more cleanly. And the token reduction is real. For deployments where SSA questions are out of scope, summaries are a viable configuration. For deployments where users may ask about things the assistant previously said, or where exact entity matching matters, raw messages are necessary.

Retrieval depth (k) and context tokens

On LME-S with the best setup (Experiment 25), k=10 is optimal, J-score peaks at 0.701 with an average context window of 3,352 tokens. At k=5, multi-session drops 31% because the relevant memory may not be in the top five results. At k=15, token count jumps 48% while J-score drops 3%, the additional memories add noise without adding signal. At k=30, J-score falls to 0.653 and latency increases by approximately two seconds.

On LME-M, k=20 is needed. On LoCoMo, raw message retrieval plateaus around k=15 with no meaningful gain from k=20 or beyond.

The context token numbers are worth examining directly. At k=5, average context is 1,688 tokens. At k=10, it is 3,352. At k=30, it reaches 9,726. The jump from k=10 to k=30 triples context cost and degrades performance. In Experiment 7, k=10 at 6,782 tokens scored higher than k=20 from the baseline at 13,087 tokens using the same prompt. More context does not produce better answers. Beyond the optimal point, it introduces noise that the model struggles to filter.

Temporal grounding

Temporal Reasoning is the weakest category across all baselines. On LoCoMo without any timestamp information, temporal scores 0.103 at baseline. The failure mode is consistent: conversations contain relative time expressions (“yesterday,” “last week,” “about three months ago”), which get stored in the database at some point and retrieved weeks or months later. When the model reads “I went to the museum yesterday,” it resolves “yesterday” against the inference clock (the current date) rather than the date of the original conversation. The memory was stored in February 2026. The museum visit was “yesterday.” The model concludes the museum visit was in February 2026.

Three interventions produced three very different results:

1. Prepend the original session date to every stored memory. When a memory block is stored, attach the date of the conversation it came from. When retrieved, the model can read “Session from 7 May 2023: I went to the LGBTQ support group yesterday” and correctly interpret “yesterday” as 6 May 2023. On LoCoMo, this single change takes temporal reasoning from 0.103 to 0.738, approximately a 7x improvement in one experiment.
2. Add “Today is {question_date}” to the answer prompt. Providing the model with an explicit anchor for when the question is being asked is the single highest-leverage change on LME-S: overall J-score rises from 0.571 to 0.637 with no other modifications. Temporal scores jump from approximately 0.18 to 0.263, the largest single-category gain from any intervention on that benchmark.
3. Hard-filter future memories. This approach feels logical: if a question was asked in October, memories from November should be irrelevant. In practice, it degrades performance. LME-S temporal drops from 0.538 to 0.398. Future memories frequently contain session summaries and contextual signals that help the model understand the arc of events leading up to the question date. Removing them removes grounding rather than noise.

Replacing relevance ranking with recency-based sorting produces a similar degradation. Overall J falls to 0.554, TR to 0.346, MS to 0.361. Relevance scores already encode temporal information, a memory from last week that is topically relevant to the query ranks higher than a memory from yesterday that is not. Overriding relevance with pure recency discards the semantic component.

BM25 reranking

We ran three BM25 post-hoc reranking experiments using different initial pool sizes: retrieve 10 then rerank to 10, retrieve 50 then rerank to 10, and retrieve 100 then rerank to 10. The results were unambiguous: pool=10 scored 0.635, pool=50 scored 0.615, pool=100 scored 0.589.

When the initial pool is small, vector search has already done the semantic filtering and BM25 sharpens keyword precision within that filtered set. When the pool expands to 100, the average relevance of the initial vector results drops, and BM25 ends up substituting good semantically-retrieved memories with ones that match the question’s keywords but are not actually relevant. Pool=100 performs worse than no reranking at all.

Hybrid search at query time, combining vector similarity, BM25, and entity extraction in a single Couchbase full-text search (FTS) call, is a different approach with different tradeoffs. The clearest gain was in Temporal Reasoning: pure vector search frequently missed specific event names or dates when the question phrasing diverged from the stored memory text. “How many weeks ago did I attend the Nordstrom friends and family sale?” can surface a memory about “I recently bought boots at Macy’s Black Friday” because they are thematically adjacent and the vector embedding averages over the whole conversation turn. Hybrid search with entity extraction matches “Nordstrom friends and family sale” as a named entity directly, regardless of surrounding phrasing. The tradeoff is Multi-Session precision: keyword matching surfaces memories from the right topic but potentially the wrong session, and the model can struggle to determine which session’s version is authoritative.

The interaction matrix

None of these decisions are independent.

Summaries improve SSP and MS while degrading SSA. Recency sorting helps SSP while collapsing TR and MS. Large-pool BM25 reranking helps SSP while hurting MS. Hybrid search improves SSU and TR while increasing MS cross-session confusion. Adding a timestamp to the prompt significantly improves TR with minimal effect elsewhere.

Every design decision has asymmetric effects across question types, and there is no universally optimal configuration. A single overall score obscures this structure entirely. The interaction matrix is the more useful output for making informed design decisions. The right configuration depends on which question types matter most for a given application.

Results

On LoCoMo, Couchbase Agent Memory achieves an 82.3% LLM Score. The largest gain is in Temporal (72.3% vs. 48.3%), which reflects the timestamp grounding work directly. Multi Hop is the strongest absolute category at 90.2%.

On LME-S with gpt-4o-mini, we achieve 69.4% overall. The gaps in different categories are consistent with the interaction matrix; configurations exist that would improve SSP and TR, but they trade away the SSA and MS leads. On LME-S with gpt-4o, we achieve 75.9% overall. 

On LME-M, we achieve 65.3% overall at 1.5 million tokens per problem. No known published comparison exists at this scale; these are the first reported results on LME-M. Single-session recall is near-perfect (SSA at 94.6%, SSU at 92.9%), confirming that retrieval works well when the answer is contained within a single session. The harder categories (MS at 48.9%, TR at 45.1%, SSP at 40.0%) reflect the cost of the 20x search space expansion.

What we learned

The core finding is that agent memory retrieval has a design space distinct from document RAG, and the right configuration depends on question-type distribution and haystack scale. This is worth stating clearly because the field tends to evaluate under RAG assumptions.

Several broader observations emerged from the experimental process.

Evaluation methodology shapes results as much as system design does. The judge prompt is not a neutral measurement instrument. It has biases toward longer answers, verbosity over precision,  and topic adjacency over factual accuracy. Switching judges shifts scores by 5-10 points. Using the same judge across all experiments within a study is a minimum requirement for interpretable results. Comparing scores across studies that used different judges is not valid.

Small-haystack optimization does not transfer to production scale. The optimal k at LME-S scale is 10. At LME-M scale, it is 20. Prompt variants that top the small benchmark underperform on the large one. Systems tuned entirely on small-haystack benchmarks may be misconfigured for the scale at which they are actually deployed.

Temporal grounding is a concrete, fixable problem. The approximately 7x improvement in LoCoMo temporal recall from a single ingestion-time change by prepending session dates to stored memories is the most actionable finding in the study. This failure mode is invisible on benchmarks without temporal questions and consequential in production when users ask time-relative questions. The fix has negligible cost and should be applied by default.

Context sufficiency separates retrieval problems from reasoning problems. Without it, a poor J-score could indicate that the wrong memories were retrieved, that no memories were retrieved at all, or that the right memories were retrieved but the model failed to use them. These require different interventions. Context sufficiency makes the distinction explicit.

The dominant temporal failure mode is structural. In 55% of temporal failures, the problem is retrieval: the question phrasing diverges from how the event was stored, the embedding averages over too much noise in a long conversation turn, and the correct memory is buried in the rankings. Timestamp grounding and hybrid search reduce the frequency of this failure but do not eliminate it. Approaches that extract structured event representations at ingestion time address the root cause more directly. This is worth further exploration.

Most teams building agent memory systems are evaluating on small haystacks, trusting a single overall score, and treating the problem like a harder version of RAG. This evaluation argues that all three of those assumptions are wrong. The failure modes are specific. The design space is real. And the right configuration only reveals itself at the scale where things actually break. The question worth asking is: if you never tested your system at production scale, do you actually know whether it works?