Anatomy of an agent memory benchmark

The claim under the benchmark

A paired benchmark looks like a measurement and is actually an argument. You run a task twice, change one thing, and report the difference. Everything the number means rests on a premise you never state out loud: that the one thing you changed really was the only thing that changed.

Here is how quietly that premise dies. You pick a real repository to benchmark against, because benchmarking on your own codebase is worthless. You pin it to a commit, package it into a container, and run one agent with your memory system and one without. The repository ships a CLAUDE.md. The harness reads it into the system prompt before the first turn, in both arms. Nothing in your task definition mentions it.

Your no-memory arm now has a hand-written memory system. Your measured effect shrinks for a reason that has nothing to do with your product.

That is the normal case, not an exotic one. Commit history is another way in. So is a stray .orig file. The word fixture in a project description tells the agent it is inside an experiment, and an agent that knows it is being measured does not behave like one doing ordinary work.

Running the agents is the easy part. The apparatus is what you build so the comparison is allowed to mean something.

stele-bench is that apparatus, now MIT. This is a walk through its architecture, the failure each piece was built after, and what you would have to write to point it at a memory system that is not ours.

The shape of a run

One paired run is three phases. A shared preparation phase that produces byte-identical inputs, a split into two containers that differ in one respect, and a shared grading phase that applies the same graders in the same order to whatever comes back.

Each stage shuts one specific way the answer can get in. Building the tree from Git objects takes the history with it. Redaction removes hand-written project memory. Certification catches leakage and anything that announces the experiment. The mutation audit catches work outside the declared outputs. The last way in is us, which is what the admission decision is for.

Contracts before anything else

The base class every artifact inherits is four lines long and is the reason preregistration is enforceable at all.

class ContractModel(BaseModel):
    """Strict base class so an artifact does not silently change its meaning."""

    model_config = ConfigDict(extra="forbid", frozen=True)

Preregistration means freezing the prompt, the verifier, the model, the budget and the analysis before any money is spent, then declaring that nothing moved. A permissive parser makes that declaration unfalsifiable. If a challenge manifest can gain a field that older code ignores, then the manifest you ran is not necessarily the manifest you froze, and no digest in the world will tell you, because the digest covers bytes while the parser decides meaning.

The same instinct runs down to individual fields. Artifact paths are validated as portable relative POSIX paths, so a manifest cannot carry an absolute path, a Windows drive letter, a backslash, or a .. component. Every digest field is a compiled pattern rather than a string. This is what keeps a frozen artifact frozen when someone runs it on a different machine a year from now.

A corpus with no history

The evaluated project tree is built from Git objects and only from Git objects. It never inherits a caller's worktree, ignored files, build output, or .git directory.

Two reasons, and the second one cost us a paid run. The first is reproducibility: a tree assembled from a checkout carries whatever that machine happened to have lying around, and a benchmark corpus that differs per operator is not a corpus. The second is that repository history answers questions. A question like why does this keep breaking has its answer distributed across commit messages, and an agent with git log can go get it. One of our early cohorts was thrown out precisely because both arms went digging in history that should never have been in the container, which made the run informative about agent behaviour and useless as a memory comparison.

If your memory system's pitch is that it holds things the code does not say, then commit history is your competitor. Remove it on purpose, from both arms, and say in the writeup that you did.

Certification, and the file everybody forgets

Certification is a static, zero-model, fail-closed gate over every surface an evaluated agent can see: the source corpus, the evidence corpus, every declared graph arm, and the agent-visible text of the challenge itself. It runs before any state is created and before any tokens are spent, which is exactly why it must not need a model to run. It classifies what it finds into five kinds.

auto-loaded-context     a file the harness reads into the prompt unasked
suspicious-metadata     .git, .env, credentials, runner-owned task wiring
benchmark-awareness     the packaged inputs announce the experiment
patch-or-diff           a .patch, .orig, or a unified diff hunk in the tree
solution-leak           a challenge's declared answer phrase, already present

The first kind is the one that is genuinely hard to see, because the file does its damage without ever appearing in your task definition.

AUTO_LOADED_CONTEXT_PATTERNS: tuple[str, ...] = (
    "AGENT.md", "AGENTS.md", "CLAUDE.md", "CLAUDE.local.md",
    "GEMINI.md", "QWEN.md", "copilot-instructions.md",
    ".aider.conf.yml", ".clinerules", ".cursorrules", ".goosehints",
    ".mcp.json", ".windsurfrules",
    ".claude/", ".codex/", ".continue/", ".cursor/", ".gemini/", …
)

A bare name matches that basename anywhere in the tree; a trailing slash matches any path component, so .claude/ covers a whole skills directory. These files are hand-written project memory. Packaging one is packaging a rival memory system into every arm at once.

The benchmark-awareness probe is where a gate like this usually goes wrong, and the interesting engineering is in what the list leaves out.

# Terms that would tell an agent it is inside an experiment. Deliberately
# narrow: each one is a harness or arm identity, not ordinary engineering
# prose, so a real repository does not trip them. Words like "condition",
# "arm", "trial", "synthetic" and "ground truth" are NOT here: they occur
# naturally in real project knowledge and would make certification cry wolf.

A gate that fires on ordinary words gets a suppression list, then a --force flag, then it is off. So the strict vocabulary is scoped rather than broadened. Harness identity terms are checked everywhere. A second, stricter set includes terms like fixture, control arm and negative control. Those are only checked against a graph's identity fields: the project, workspace and branch names the benchmark writes itself. They are never checked against node bodies.

That last part matters. A graph backfilled from a real project says spin up a temp directory with test fixtures in ordinary engineering prose. Flagging that is noise, and noise is what trains you to ignore the gate.

One more rule, which sounds pedantic until it saves you. A missing corpus, fixture artifact, or graph is an error, never a skipped check. A gate that quietly covers nothing still reports success, and that is worse than not having one. The integration test that certifies the real pinned corpus fails rather than skips when the checkout is absent, and the opt-out is an explicit environment variable that says in its own name that the corpus went unchecked.

Repair and inspection are different jobs

A pinned corpus belongs to somebody else. You cannot ask an upstream project to delete its CLAUDE.md so your benchmark works, so redaction has to exist. The design decision is that it is the only thing allowed to modify a packaged tree, and it is a separate function from the gate that inspects one.

Redaction removes auto-loaded context while packaging, identically for every arm, and writes a receipt naming each removed path with its digest. Certification then runs over the result and only ever reads. Because the two are separate, an unredacted or hand-edited tree cannot reach an agent: there is no code path where the gate notices a problem and fixes it. A gate that repairs what it finds is a gate you can never trust to have found nothing.

The boundary that makes it yours

Everything so far is generic. The part that decides whether stele-bench is a benchmark or a Stele benchmark is one adapter, and it is the thinnest component in the system.

You configure an argv prefix and a fixture root. The runner executes <prefix> restore, snapshot, or destroy through create_subprocess_exec, with no shell anywhere in the path. It writes exactly one compact JSON object to stdin and expects exactly one JSON object on stdout. Stderr is reserved for bounded operator diagnostics and is never parsed as data, so a chatty provisioner cannot accidentally become a data channel.

What it refuses is part of the contract too. It rejects a fixture artifact whose resolved path escapes the fixture root, an artifact whose SHA-256 does not match the manifest, a response that is not a single JSON object, a response carrying unknown fields, a non-zero exit, a timeout, and output past a byte limit. Each of those is a way a provisioner can be subtly wrong while looking fine.

The three verbs are the whole integration surface. To benchmark a different memory system you write one executable that speaks them:

$ your-provisioner restore   # stdin:  { fixture, graph_arm, run_identity,
                             #            artifacts: { evidence_corpus, graph } }
                             # stdout: { project, queue_before_count, restore_seconds }

$ your-provisioner snapshot  # stdin:  { project }
                             # stdout: { state, probes, queue_after_count, snapshot_seconds }

$ your-provisioner destroy   # stdin:  { project }
                             # stdout: { "ok": true }

Restore takes a frozen memory state and makes it live for one run. Snapshot independently re-reads what is there afterwards, including named full-text, semantic and graph retrieval probes, so the run can prove the memory it was supposed to have was actually present and retrievable rather than assumed. Destroy tears the state down in a finalizer.

The snapshot receipt matters more than it looks. Without it, a treatment arm that silently restored an empty graph produces a perfectly clean null result, and you publish memory did not help when the truth is that memory was never there.

A challenge and a fixture are different objects

A challenge owns the question: the visible prompt, the allowed output paths, resource limits, and a digest-pinned verifier. A fixture owns the evidence: the frozen corpus and the graph state handed to one condition. They are independent on purpose, so the same question can run against a no-memory arm and a restored graph without the condition ever appearing in anything the agent can read.

{
  "schema_version": 1,
  "challenge_id": "settings-label-roundtrip",
  "prompt": "On the Settings screen, rename the row currently labeled
             \"Check for Updates\" to \"Automatic Update Checks\" …",
  "allowed_output_paths": [
    "/workspace/src/screens/settings.rs", "/workspace/tests",
    "/workspace/Cargo.lock", "/workspace/target"
  ],
  "agent_timeout_seconds": 900,
  "environment": {
    "network_mode": "public", "cpus": 2, "memory_mb": 4096,
    "prebuild_argv": ["cargo", "test", "--no-run", "--lib"]
  },
  "verifier": {
    "kind": "script", "runtime": "posix-shell",
    "asset": { "path": "verifier.sh", "sha256": "7aba6b60…" },
    "timeout_seconds": 900
  }
}

The verifier is runner-owned. It lives outside the evaluated project tree and is never included in the prompt, so grading a task and performing it cannot see each other. Resource limits live in the manifest because they are part of the experiment. CPU count, memory, and whether the project was prebuilt all change the task. Two of our early runs were invalidated by exactly that.

The catalog spans several kinds of memory value deliberately, rather than repeating one flattering pattern. Investigation questions grade whether the agent reaches a grounded answer and how much source exploration it needed. Focused coding questions exercise hidden cross-file contracts. Continuity questions use a follow-up prompt, which produces a two-step task sharing one verifier, scoring only the final step, with the agent resuming its own native session across the boundary. And some challenges are low-memory controls, chosen because memory should not help, so the suite measures its own overhead instead of quietly excluding the cases that make it look bad.

Grading in three layers, and who wrote the sentence

A single pass or fail cannot tell you whether the memory was delivered, whether the agent used it, or whether it changed the result. So every run reports those three separately, and none of them implies the next. A correct treatment arm with delivered memory is not evidence that memory caused the success, because the control may well have succeeded too.

The first layer is where naive implementations quietly cheat. The obvious way to detect delivery is to search the transcript for a distinctive phrase from the memory. The problem is that a transcript contains the agent's own words.

# A match on an "agent"-sourced step never counts: the agent authoring
# that exact phrase is not evidence the memory system delivered it.
_NON_AGENT_SOURCES = frozenset({"user", "system"})

So only non-agent steps are eligible, and on the harness we run against, injected context arrives as a specific attachment type carrying a provenance comment that names the channel it came through. Prompt-time retrieval and anchor-triggered recall are two different mechanisms, and naming the channel keeps them apart. A plain delivered-or-not boolean would report them as the same thing.

The mutation audit, and the lockfile that voided a run

Before the challenge verifier runs, a hidden generic audit compares the finished workspace against a snapshot taken before the agent started. Anything added, removed, changed, permission-modified, symlink-retargeted or type-swapped outside the declared output subtrees forces reward zero. The receipt is deterministic and stays in the runner-owned area, outside anything the agent can see or edit.

This is the check that catches an agent solving the task by editing the test, and it is also the check that will fail you for a reason you did not think of. One of our runs was excluded because ordinary cargo execution wrote a lockfile that the allowlist did not declare. The agent did nothing wrong. The toolchain did what toolchains do, the audit correctly reported an undeclared mutation, and the run was thrown out.

The lesson generalises past Rust: your output allowlist has to describe what your toolchain writes, not what you intend to write. That is why the manifest above lists Cargo.lock and target alongside the source file the task is actually about.

The money fence

A paid agent benchmark spends real money on a shared balance, and the usual protection, a per-run cost cap, does not protect the thing you care about. A cap bounds what one run may spend. It says nothing about what is left for everything else.

We know that because of an incident the fence is named after: a run that stayed comfortably under its own budget and drained the shared balance anyway, which on our setup means taking the production assistant down with it. So the preflight now checks four things in order, before a paid pair may start.

1  key identity     the exported key matches a configured SHA-256 fingerprint,
                    so a run cannot be billed to the production key by accident
2  per-key budget   the quota API is asked directly whether the eval key really
                    has an external ceiling; an absent budget fails
3  shared headroom  team balance must cover a production reserve PLUS the
                    acknowledged pair cap
4  observed drain   between arms, re-read the balance and stop if more drained
                    than was declared

Step four exists because the first three prove the pair fits, and none of them can prove the pair behaves. It also deliberately over-attributes: observed drain is the drop in the shared balance, so it counts spending by anything else on the account too. Whoever is spending it, the next dollar past that line comes out of the reserve, and stopping is the right response either way.

The two failure modes are deliberately asymmetric. A transport error on the balance check fails open, and records that it was unobserved. One flaky GET should not abandon a pair that has already been paid for, and a check that did not run is fine as long as the record says so. A rejected key fails closed. The preflight already proved that key was good, so a 401 now means it changed underneath the run, and continuing would spend against something nobody verified.

Statistics for small samples

Agent benchmarks are expensive, which means small samples, which is exactly where the textbook approximations misbehave. Everything in the statistics module is exact: integer combinatorics, no sampling, no normal approximation to a binomial tail, no dependency beyond the standard library. The same inputs always produce the same numbers, so a receipt can be re-derived from its own recorded attempts.

Three of the rules are worth repeating whatever you build on. Do not use a Wald interval at 0/n or n/n, where it collapses to zero width and claims certainty from five observations. Use paired methods, because the arms ran the same tasks and a hard task is hard for both; subtracting two independent intervals throws that away and reports a range far wider than the data supports. And compute no significance test at all for a continuous measure below five attempts. Tokens and wall time at that sample size are noise wearing a decimal point, so the summary says exploratory and stops there.

What makes two runs comparable

Regression detection means plotting one question's score across runs, and that only means something when the runs asked the same question of the same model against the same memory. Change the verifier and the score moves without memory having changed at all. So every durable record carries an explicit comparability key: challenge, fixture, model, both conditions, the manifest and verifier digests, and the source commit. The viewer refuses to put two different keys on one line.

One input is deliberately left out of that key, and it is the build under test. That build is the thing whose effect on the score you are trying to see. Put it in the identity and every build becomes its own incomparable island, which turns regression detection off exactly when it would have told you something. It is recorded against each data point instead, so a step in a trend line can be traced to the build that caused it.

Admission is a decision, not a status

The final source of bias is us, and it is the hardest one to close with code. So the apparatus does the one thing it can. It refuses to let a finished run count as an admitted one.

A run that finishes is completed. Whether it may contribute to a published number is a separate, recorded decision with its own reason code. Excluded runs are retained in full, never repaired and never reinterpreted. Cleanup may reorganise files or redact host paths and credentials; it may not remove a run because its direction is unfavourable.

Preregistration versioning enforces the same discipline over time. An apparatus correction does not fix the old run, it creates a new preregistered version, and the earlier outcome stays in the record as excluded. That is why the published cohorts carry version numbers: a question that reached v4 is a question that failed three times in ways worth reading about, and the record says so. A publication test in CI checks that every admission has a matching durable record, that the manifest covers every published file with a digest, and that no host paths or secret-shaped strings made it into the release.

Pointing it at your own system

The repository is MIT and contains the full harness, the challenge definitions and their verifiers, the preregistrations, and every run we have published, accepted and excluded alike, with the trajectories and a manifest hashing every file.

To measure a different memory system you write one executable that speaks three verbs over JSON, and declare it as an argv prefix. The runner never learns your schema. That is what makes the harness portable instead of Stele-shaped. Everything else, the certification, the tare challenges, the exact tests, the admission split, applies unchanged, because none of it was ever about our product.

The preliminary results we ran through it are published separately, with every admitted and excluded run attached: when does project memory help a coding agent. Read the apparatus first. It is what makes those numbers worth reading.