When a fact stops being true

The problem with being right later

In March the team decides sessions live in a signed cookie. It goes into the record with the reasoning attached: no server-side session store to operate, no extra round trip on every request.

In August they reverse it. Revocation turned out to matter more than the round trip, and you cannot revoke a signed cookie without the store you were avoiding. Sessions move into the database.

There are now two questions, and most systems only handle the first.

1. What is true now? Sessions are in the database. Any agent asking about session handling has to get this answer and not the other one.
2. Why did this change? Six months from now somebody proposes signed cookies again, for the reasons that were good in March. Without the history that proposal looks new. With it, it looks like a settled argument whose counter-argument is already written down.

Delete the old decision and you answer the first question perfectly while destroying the answer to the second. Keep both as equals and the agent gets two contradictory facts with the same standing, which is worse than either one alone. Neither is acceptable, which is why retiring a fact is its own operation rather than an edit in place.

What it looked like at first

The first implementation was the obvious one. Recording that B supersedes A drew an edge from B to A and archived A. Archived nodes were excluded from recall. The old fact stayed in the database and stayed visible in the web UI if you went looking for it, but as far as any agent was concerned it was gone.

On paper that is correct. In practice it has one property that turned out to outweigh everything else: a mistaken retirement is unrecoverable in the only place that counts.

If an agent wrongly decides that a new fact retires an old one, it has not mislabelled anything. It has removed a true fact from every future agent's world, silently, with no error raised and nothing to notice. The next agent does not encounter a wrong answer. It encounters no answer, which is indistinguishable from the project never having decided at all.

The number that did not move

We would not have prioritised this if it were a theoretical risk. Here is what it actually looked like.

We were testing how well agents fold new source material into an existing graph: hand one a meeting transcript or a design document, have it work out what the project already knows, and record what changed. Retiring the facts that the new material genuinely reversed is precisely the job.

On conversational source material, agents wrote a false retirement eleven times out of eleven.

Not eleven mistakes over a long run. Every single opportunity. And the mistakes were reasonable. Conversation is full of sentences that sound like reversals and are not: someone floats an alternative, someone objects to a decision without overturning it, a plan gets refined rather than replaced. A model reading a transcript sees a new statement about a topic it already holds a fact for, and the shape of new statement about known topic sits very close to the shape of a genuine reversal.

So we did the normal thing. We rewrote the guidance. We spelled out when a retirement is warranted and when it is not, gave worked examples of the near misses, and put the warning at the point of use rather than in a document nobody opens.

It moved the rate by zero. Still eleven out of eleven.

That result is worth sitting with, because the instinct after a guidance change fails is to go and write better guidance. What it actually tells you is that the error is not caused by the agent being under-informed. It is caused by the task being genuinely ambiguous at the point of decision, and no quantity of instruction resolves an ambiguity that lives in the source material rather than in the reader.

Making the failure survivable

The fix was to stop hiding retired knowledge from recall entirely. A retired fact comes back, marked as retired, ranked below whatever replaced it.

That one change converts the failure mode:

• Before. A wrong retirement destroys a true fact. Silent, unrecoverable in the surface that matters, and detectable only by noticing an absence, which nobody does.
• After. A wrong retirement puts a wrong label on a true fact. The fact still surfaces, it is still readable, the label is visible, and anyone who spots it can reverse it.

A wrong label is an ordinary bug. A destroyed fact is a data loss event. Trading the second for the first is worth a fair amount of ranking complexity.

It also made the system finally answer the question that justified having a retirement mechanism at all. If the old decision is unreachable, what did we believe before and why did it change has no answer, and you have built an elaborate apparatus to achieve exactly the same result as editing the fact in place.

Rank, do not hide

Once a retired fact is allowed back into the candidate pool, the question becomes how hard to push it down. Too gentle and a dead decision crowds out the live one, which is the failure the feature exists to prevent. Too harsh and it never appears even when somebody is explicitly asking about history.

What we settled on is a set of multipliers applied wherever recall ranks. They are deliberately blunt:

retired knowledge      × 0.35   needs a far stronger match than a live fact
done task, fresh       × 0.60   finished this week: this is the changelog
done task, decayed     → ~0.2   finished months ago: this is exhaust
cancelled task         × 0.15   the harshest discount in the system

A retired fact at 0.35 has to be a substantially better match than any live fact before it outranks one. It surfaces when it genuinely is the best answer, which is to say when somebody asked about the past, and not merely when it is topically nearby.

Cancelled work gets the harshest treatment of anything in the system, and the reason is worth spelling out. A cancelled task is imperative text describing work the team deliberately decided not to do. An agent that retrieves Add SSO login for enterprise accounts without the cancellation attached will go and implement it. That is an argument for the marker and for the floor of the ranking. It is not an argument for hiding, because hiding destroys the answer to did we already consider this?, and that answer saves real time.

The rule that came out of it

There was a second version of the same mistake sitting in the codebase, and we only found it because of the first one.

Completed tasks were hidden from recall, on the theory that finished work is exhaust. Then a case turned up that broke the theory cleanly. A task had been completed four hours earlier. Its completion note was the only written explanation anywhere of how the feature worked. A question about that exact feature could not reach it, while plain keyword search found it immediately.

The model was missing time. A task finished this morning is the changelog. A task finished eight months ago is exhaust. They carry the same status and completely different value, so status was never the right thing to be deciding visibility with.

Status governs rank and label. It never governs existence.

Everything reachable stays reachable. What changes is where it lands in the ordering and what label it arrives carrying. There is one deliberate exception, an explicit hide flag a person sets on purpose, and being explicit is the entire point of it.

What the agent actually receives

The marker matters as much as the ranking does, because an unlabelled retired fact is worse than no fact at all. When a retired decision does surface, it arrives with its status, its reason, and a pointer to whatever replaced it:

RETIRED · decision · "Sessions in a signed cookie"
  replaced by: "Sessions in the database, revocable"
  reason:      revocation needs a server-side store; the round-trip
               cost was the wrong thing to be optimising
  written 2026-03-11 · retired 2026-08-02

ACTIVE  · decision · "Sessions in the database, revocable"
  Every session row is revocable individually. Logout and security
  response both need that; the cookie could not provide it.

The replacement is always recalled alongside the fact it retired. Surfacing a retirement on its own would leave an agent knowing the old answer is dead without knowing the new one, which is the worst of the three available states.

Every retirement also has to state a reason. Not for bookkeeping: the reason is the part a human reads six months later when the same idea comes back around, and demanding it at write time is the only reliable moment to capture it, because it is the only moment anybody still knows it.

Where this leaves the false retirements

Honestly, they still happen. We did not fix the classification problem, because we do not believe guidance can fix it, and we have the negative result to back that up.

What changed is the cost. A wrong retirement is now a visible wrong label on a fact that still shows up, which the health check flags as a decision retired but still doing work, and which any human reading the record can reverse in a moment. It is a nuisance rather than a loss.

There is one test case we care about more than the rate itself. Take a decision that was later retired, then ask an agent to do something the old decision forbade and the new one allows. The correct behaviour is to get on with it without objection. An agent that pushes back is one that treated a retired fact as live, and that is a bug with a real cost to the person on the other end of it. It is the sharpest available test of whether retirement works at all, because it is the only case where a false positive is the failure rather than the safe option.

The broader lesson has outlived the feature. Nearly every time we reached for hiding something, downranking it with an honest label turned out to be the better call. Hiding destroys an answer. Downranking preserves the answer and stops it from getting in the way, and those were the two things we actually wanted all along.