What We Learned Evaluating Agent Memory:The Setup (Part 1) 

Agent memory is one of the harder infrastructure problems in applied AI right now, and the benchmark ecosystem around it is still taking shape. This is a two-part write-up. Part 1 covers what we were measuring against, how we measured it, and how the pipeline is built. Part 2 covers what the experiments actually showed. We are publishing both because the decisions that go into evaluation are as consequential as the system decisions, and that part rarely gets written down.

What is agent memory, and why is it not just RAG?

Agent memory is what makes an AI agent not feel like it has amnesia.

Traditional AI agents are stateless. Every conversation starts from zero. For example, a customer support bot handles a ticket, the user calls back two days later, and the bot has no record of the previous interaction. A coding assistant explains the same concept twice in the same week because it has no memory of the first conversation. A shopping assistant recommends something the user has already purchased.

Agent memory addresses this by maintaining a persistent layer that stores what an agent learns about a user across sessions and makes it retrievable in future interactions. The agent learns today that a user is vegetarian and allergic to shellfish, and draws on that context in October when the user asks for dinner recommendations.

If you have worked with retrieval-augmented generation (RAG) before, this might sound familiar. The surface-level mechanics are similar: store things in a vector database, retrieve relevant chunks at query time, and pass them to an LLM. But agent memory is a fundamentally different problem in three ways.

First, the corpus is personal and dynamic. In document RAG, the corpus is usually a static collection, a knowledge base, a set of PDFs, and a codebase. In agent memory, the corpus is a user’s entire conversational history. It grows with every session and is different for every user.

Second, the queries are autobiographical rather than informational. A knowledge base search is a query about the world. A memory query is a query about the user, things they said, preferences they expressed, and experiences they mentioned. “What’s the capital of France?” and “What did I say about my dietary restrictions?” are not the same retrieval problem.

Third, time is a first-class dimension. The same fact can appear in contradictory form across sessions, and recency determines which version is correct. If a user mentioned two cats in March and three cats in June, the correct answer to “How many cats do I have?” depends entirely on when the question is being asked. A document retrieval system does not need to reason about this. An agent memory system does, constantly.

This is why a RAG system cannot simply be relabeled as agent memory. The design decisions are different, the failure modes are different, and as we found out, the benchmarks are different too. Document retrieval systems are evaluated on whether they can find the right chunk in a fixed corpus. Agent memory systems need to handle a corpus that is personal, growing, and full of contradictions. The benchmarks that exist for RAG do not surface the failure modes that matter for agent memory. We had to find the ones that do.

The benchmarks: what exists and why it matters

Before measuring anything, we had to decide what to measure on.

The agent memory benchmark landscape is still relatively young. We worked with two of them: LoCoMo and LongMemEval.

LoCoMo (Long-term Conversation Memory) is built around 10 long conversational logs – naturalistic conversations between two people that meander, reference past events, and carry implicit context. From those 10 logs, the benchmark generates 1,540 questions across four categories:

• Single Hop: One piece of information, directly stated somewhere. “What did Caroline research?” -> “adoption agencies.”
• Multi Hop: The answer requires connecting information from two or more places. “What did the charity race raise awareness for?” -> you have to find both that there was a charity race and that it was for mental health.
• Temporal: Questions about when things happened. “When did Melanie paint a sunrise?” -> “2022.”
• Open Domain: Questions that require reasoning rather than retrieval. “Would Caroline still want to pursue counseling if she hadn’t received support growing up?” -> the answer is not stated anywhere, it has to be inferred.

LoCoMo tests long-horizon naturalistic recall. The conversations are messy in the way real conversations are messy. Answers are embedded in context rather than neatly labeled.

LongMemEval is built differently. It is 500 questions, each placed in a haystack of conversation sessions. The question types are more granular and, from a system-design perspective, more revealing:

• Single-Session User (SSU): Something the user said in one session. “How long is my commute to work?” -> “45 minutes.”
• Single-Session Assistant (SSA): Something the assistant said in one session. “Can you remind me of the restaurant in Rome that I liked?” -> “Roscioli.” This category matters because the answer lives in the assistant’s own prior output, not the user’s.
• Single-Session Preference (SSP): A preference the user expressed. “Can you suggest accessories that complement my photography setup?” -> the system needs to have retained the user’s preference for Sony-compatible gear.
• Multi-Session (MS): The answer requires synthesizing information from multiple different sessions.
• Knowledge Update (KU): The information has changed. “How many bikes do I currently own?” -> “4.” But earlier in the history, the answer was 3. The system has to recognize that information has been overwritten.
• Temporal Reasoning (TR): Questions requiring time arithmetic. “How many months since my last museum visit?” -> the system has to identify when the visit occurred, when the question is being asked, and calculate the difference.

The Knowledge Update category is particularly well-designed because most benchmarks do not test whether a system handles contradictions in conversation history. An agent memory system that cannot handle KU will confidently return outdated information, which is one of the more consequential failure modes in production.

LoCoMo vs. LongMemEval: what makes them genuinely different

These two benchmarks test different things, and understanding the difference matters before looking at results.

LoCoMo uses real, unscripted conversations between two people. The memory system needs to handle both sides, what Caroline said and what Melanie said. The ingestion pipeline reflects this with a dual point-of-view extraction: each exchange creates two memory blocks, one keyed to each speaker. This is not just a formatting convention. LoCoMo questions can be asked from either perspective, and the system needs each speaker’s version of events available at retrieval time.

LongMemEval uses conversations between a user and an AI assistant, with questions asked by the user about their own history. In the LME-M setup, we also use dual-POV (one user-perspective block and one assistant-perspective block per exchange), because the SSA question type specifically asks what the assistant said. In LME-S, we simplified to a single sequential POV and achieved identical scores at half the latency. That tradeoff is covered in Part 2.

The other structural difference is scale. LoCoMo has 10 conversations. LongMemEval has 500 questions, each with its own haystack. LoCoMo tests depth of recall over a single extended relationship. LongMemEval tests whether the retrieval system can surface the right memory across a large and growing collection of sessions.

The two scales of LongMemEval, and why this matters

LongMemEval comes in two variants: LME-S and LME-M. Same 500 questions, same question type distribution. The difference is how hard the system has to search.

LME-S places each answer in a haystack of roughly 46 sessions (approximately 115,000 tokens per problem). LME-M places the same answer in roughly 500 sessions, approximately 1.5 million tokens per problem. That is a 20x expansion of the search space with the same needle to find.

This turned out to be one of the most important findings of the project. The configuration that works well on LME-S does not transfer to LME-M. If a system is tuned on LME-S and deployed at production scale, it is likely misconfigured in ways that are difficult to catch without evaluating at both scales.

The clearest example is retrieval depth. The optimal k on LME-S is 10. At LME-M scale, k=20 is needed. With more sessions in the haystack, the relevant memory is more diluted, and a deeper retrieval pool is required to reliably surface it. Temporal Reasoning, which shows almost no sensitivity to k on LME-S, benefits meaningfully from higher k on LME-M, because the relevant date context is spread across more sessions.

Prompt variants diverge as well. The answer prompt that performs best on LME-S performs measurably worse on LME-M. We observed an 11.8-point gap between prompt variants on LME-M (72.4% vs. 60.6%).

LME-S is a useful development benchmark. LME-M is closer to what production actually looks like. Evaluating on both, and paying attention when the results diverge, is worth the effort.

How we measured performance

We tracked three metrics across all experiments.

BLEU score counts n-gram overlaps between the generated answer and the reference answer. For a simple example: If the expected answer is “bob likes pancakes” and the system returns “bob likes pancakes, cakes, and mustard,” BLEU scores 0.4 because only two of the five word pairs in the output match the expected answer. BLEU is a useful sanity check but penalizes paraphrasing and rewards verbatim matches in a way that does not always reflect whether an answer is substantively correct.

F1 score measures word-level overlap without requiring word order, computing precision (what fraction of generated words appear in the reference), and recall (what fraction of reference words appear in the output). For the same example, F1 scores 0.67. Better, but still purely lexical.

LLM Judge (J-score) is the primary metric. An LLM reads the expected answer and the generated answer and judges whether the generated answer is semantically correct. For “bob likes pancakes” versus “bob likes pancakes, cakes, and mustard,” the judge scores 1, the generated answer contains the correct information even with additional content. This is the metric we trust most for assessing whether the system is actually working.

One important note on J-score: the judge prompt matters as much as the system being evaluated. LoCoMo’s judge is configured to be generous – “as long as the answer touches on the same topic, mark it correct.” LongMemEval uses category-specific variants: one for temporal reasoning (allows off-by-one errors in day counts), one for knowledge update (marks the updated answer correct even if old information is also mentioned), one for preferences (rubric-based), and one for straightforward accuracy. Running our system with a much stricter judge drops scores by several points independently of any system change. Cross-judge comparisons shift scores by 5-10 points and are not interpretable. When we ran LoCoMo with judge prompts used by other memory solutions, we obtained a theoretically impossible 100% score, which immediately indicated the comparison was inflated by the prompt rather than reflecting retrieval quality. Reporting a J-score without specifying the judge prompt is like reporting a test score without specifying which test was administered.

How the pipeline works

When a conversation turn enters the system, it goes through a dual point-of-view extraction before anything is written to the database. The ingestion code pairs each user message with the next assistant response and creates two memory blocks, one keyed to the user, one keyed to the assistant. “User said X, Assistant said Y” produces one block from the user’s perspective and a separate block from the assistant’s.

With context_required=True configured on every pipeline, the system fires an LLM call during ingestion to extract a prose summary and a list of discrete facts from each exchange. What gets written to Couchbase are three things: the raw message text, the summary, and a contexts list of fact strings. Timestamps are attached as annotations in human-readable format (“1:10 pm on 27 March, 2023”) so the model can reason about them naturally at retrieval time.

A memory block looks roughly like this:

{
  "message": "User: When did I start the job? Assistant: You mentioned starting in March 2023.",
  "summary": "User is asking about their job start date; assistant confirmed March 2023.",
  "contexts": [
    "User started their job in March 2023",
    "User expressed uncertainty about the exact start date"
  ],
  "annotations": {
    "time": "1:10 pm on 27 March, 2023",
    "conversation_id": "abc123"
  }
}

The ingestion call returns in approximately 251ms at p50, covering validation, embedding generation, and the Couchbase write. The LLM context extraction (summary and contexts list) happens asynchronously in the background and takes approximately 3.6 seconds on average. The caller does not wait for it. This decoupling, via a dispatcher queue that separates ingestion latency from extraction latency, is what makes the system practical at scale. A synchronous mode is available when immediate searchability is required, at the cost of the 3.6 second wait.

Search works differently. The get_memory() call embeds the query (approximately 305ms at p50) and runs a vector similarity search against stored blocks in Couchbase (approximately 5ms at p50). Total client-facing latency is around 298ms at p50. The bottleneck is the embedding step, not the retrieval itself. Couchbase full-text search (FTS) is fast.

For the evaluation pipeline specifically: memories are ingested once and persist in Couchbase. The search and evaluation stages run against the live database with different configuration parameters, with no re-ingestion needed between parameter sweeps. 

The pipeline is built. The benchmarks are selected. The metrics are defined. All of that is groundwork. The real question is what actually happens when you run the system against real benchmarks at scale, across multiple different experimental configurations. That is what Part 2 is about.