How do you measure whether agent memory actually helps?

The demo problem

Every agent memory product, ours included, can produce the same demo. A fresh agent gets a question, flounders. An agent with the memory gets the same question, answers it. The audience nods.

The demo is worthless, and it is worthless for a reason that took us embarrassingly long to state precisely: the demo is chosen after the fact from the cases where it worked. Nobody films the run where the memory arrived and the agent ignored it. Nobody films the run where the control read a well-commented file and got there first.

So we set out to build something that could return an answer we did not want. This is a report on the method, not on results. We do not have results worth publishing yet, and the reason we do not is the interesting part.

We cannot benchmark on our own repository

The first finding was the one that invalidated the most work.

An agent working inside our own codebase is not a normal agent. It reads our documentation, sees the tooling, and works out within a couple of turns that the memory system is the subject of the exercise. It then pays the graph conspicuous, deliberate attention. On an unrelated project with a good record, the same agent uses the same feature far more quietly, and barely narrates it.

Those are two different behaviours, and only the second one is what a customer experiences.

This is demand characteristics, the same effect that makes a participant who has guessed the hypothesis useless as a subject. It has a consequence we now treat as a hard rule: the corpus runs on an external pinned repository, never on ours.

It has a second consequence that is harder to live with. Self-observation is never a result. A session where we watch our own agent use the record beautifully is an anecdote, and it is precisely the class of anecdote the benchmark exists to replace. Sample size one, unblinded, and the observer is the most motivated party in the room.

The run that was perfect and meant nothing

Our clearest lesson came from a run that went entirely to plan.

Ten arms, all clean. Every cost fence held. Total spend about a dollar sixty. Deterministic grading, durable receipts, no infrastructure failures. By every operational measure it was a success, and the number it produced was meaningless.

The question had two reasonable readings. Under one reading you needed the stored fact to answer. Under the other you did not. The agent picked a reading, and our grader dutifully measured which reading it picked. We had built an extremely reliable apparatus for measuring the wrong thing.

That reframed the whole project. The apparatus was never the hard part. Running paired arms with certification and a money fence is engineering, and engineering is tractable. The questions are the science, and the questions are where everything goes wrong.

Eight rules a question has to pass

What came out of that is an admission checklist. A candidate question has to clear all eight before it costs any money. Three of these have each personally invalidated a paid run.

1. Unguessable

The answer must not be derivable from the repository at any token budget. The strongest form is a decision not to build something, because absent code is indistinguishable from code nobody got round to writing.

2. Unavoidable

A correct answer must be impossible without the memory. This is the one the run above died on. The test to apply before spending anything: could an agent that fully holds this memory still answer well without using it? If yes, the grader is broken before it runs.

3. Unambiguous

Exactly one reasonable reading. Ambiguity does not average out across attempts. It becomes the thing you are measuring.

4. Never “how does X work”

Frontier agents open the file and answer. That is reading comprehension, and it tells you nothing about memory.

5. Prefer questions whose answers live in history

Why do we have X. Did we already decide this. What breaks if I change this. The governing principle is distance: recall beats reading precisely when the answer is far from the question. When the answer sits in a comment above the function, the two are a wash.

6. Grade behaviour, not vocabulary

Checking whether the agent emitted a particular phrase measures parroting. Grep for a marker only where the marker is genuinely unavoidable, which is rule 2 again wearing a different hat.

7. Match the question to the channel you are testing

This one is specific to how retrieval is wired, and it is easy to get wrong. Anchor-triggered recall fires when a file is edited. It does not fire when a file is read. So an advice-shaped question can only ever exercise semantic recall, and if you wanted to know whether anchors work you have learned nothing. Testing anchors requires a task that actually edits something.

8. Prove the trajectory hits the anchor before paying

Run the control unpaid first and watch where it goes wrong. The likely wrong edit has to land on a file the tested fact is actually anchored to, and the fact has to directly correct that specific mistake.

We learned this by skipping it. We built a probe where the memory was real, the anchors were real, and the agent made its wrong edit in a different file entirely. The warning could not fire, and if it had fired it would not have addressed the mistake. Paid, and worth nothing.

Three layers of grading, reported separately

A single pass or fail number hides where the effect actually lives. We report three, and the gaps between them carry more information than any one of them.

injected          did the memory reach the agent at all?
                  read from the retained session transcript

observably used   does it appear in the answer or the reasoning?
                  a LOWER BOUND, never a count

scored            did it satisfy the grader?

On the run described above these read five, then three, then one. The interesting question was never the one. It was the two places the number dropped, which are entirely different problems: retrieval reaching the agent and being wasted, versus retrieval being used and still not producing a better answer.

The middle layer needs its qualifier stated every time. It is a lower bound, not a count. A model can read a fact, internalise it, and produce a better answer without ever referring to it. So the honest phrasing is injected, with no observable trace. Writing injected and ignored claims knowledge of something unobservable, and it happens to claim it in the direction that flatters nobody, which is not the same as being conservative.

Rules we hold ourselves to

No model as judge. Graders are deterministic. A model scoring another model's output introduces a second unvalidated system into the measurement, and when the result disagrees with expectation you cannot tell which one was wrong.

Continuous measures need far more attempts than binary ones. At five attempts, every continuous measure we tried, tokens, wall time, tool calls, was indistinguishable from noise. If the headline is cost, budget for twenty or more, or use a paired design.

Pre-register the prediction. Including for the results that would be bad for us. The most important number we do not have yet is the pure overhead of installing memory on a project where it cannot possibly help: an empty graph, a task with no history to draw on. It is the first thing a skeptic asks about, and five percent overhead and sixty percent overhead are very different products.

The experiment that actually tests the claim

Everything above measures a component. Only one design tests the product thesis, which is that the overhead pays for itself later.

It takes two sessions. Session one: both arms do a task, and the memory arm writes whatever it writes, unprompted, using only the shipped guidance. Session two: a related task requiring transfer, where the memory arm inherits the record session one produced and the control starts fresh. You measure session two, and cumulative cost across both.

The claim makes a sharp, falsifiable prediction. Session two cost inverts, and cumulative cost crosses over at some number of sessions. That crossover number is the honest headline for a product like this, and it is the one we want before we start quoting anything.

Why the shape is not obvious

It is tempting to model the overhead as one thing that decays: you pay to fill an empty graph, the graph fills, the payments stop. That model predicts the line falls and keeps falling.

The overhead is more likely two things with different behaviour. Some of it is fixed per session: tool schemas in the context window, hooks running, a search helper starting up. That cost arrives every session no matter how mature the record is, and it never amortises. The rest is write cost, which plausibly does decay as the project's durable facts get written down once.

If a meaningful share is fixed, it sets a floor the line cannot go below, and the steady state is not a percentage of the baseline. It is baseline plus fixed overhead minus whatever recall saves. Whether that lands above or below the baseline is the sign of a difference nobody has measured, which is a more precise way of saying we do not know where the crossover is, or whether there is one.

We have a reason to take the fixed half seriously. An earlier harness-level measurement showed one arm using roughly three times the tokens of the other, which looked damning until we looked at where they went: almost all of it was cached schema and helper startup, front-loaded rather than spent on extra work, and net correctness was unaffected. Raw token count partly measures how much apparatus is attached rather than how much thinking happened. It is why the primary cost measure here is the authoritative gateway receipt, with token counts kept as a cross-check and never substituted for it.

One constraint on that design is load-bearing and easy to violate. Session one's writes have to come from the shipped guidance, with nothing added. The moment the harness tells the agent what to record, the benchmark is measuring whoever wrote the prompt rather than the product.

Where this leaves us

With an apparatus and without a number, which is the right order. Pinned corpus, frozen graph, sealed container, deterministic oracle, nine gates before a token is spent, and a pre-registration locked before results exist. Three admission runs have been discarded rather than reported, each because it caught a defect that would otherwise have shipped as a number.

We could publish the pilots that came out favourably. That is exactly the demo this post opened by complaining about, and it is the reason we built the harder thing instead.

What we will say is what we would want said to us if we were evaluating this category:

• Ask what the control arm was. No memory, memory but no auto-injection, and no product installed at all answer three different questions.
• Ask how many attempts. Continuous measures at n=5 are decoration.
• Ask whether the benchmark ran on the vendor's own codebase. If it did, the agent knew it was being watched.
• Ask for the overhead number on a project where the memory cannot help. Everyone has that number. Almost nobody publishes it.

We will publish ours when the questions are good enough that we would believe the answer if it came out against us. As far as we can tell we are the only people in this category building the apparatus to find out, which is a strange thing to be able to say about a field selling measured improvements.