DRAFT Preliminary, not yet announced. The stable findings are the floor (0/6, zero documents read) and the adoption cells; Gemini numbers vary and the Opus 5 rate is a point estimate with a wide interval. Shared for early feedback.

Governed memory · stale-context lab

Your coding agent is working from outdated information, and it will not check

A coding agent with a project summary in its context trusts that summary completely. It does not verify whether the summary is still true, because a file that already answers the question supplies no reason to doubt. We reproduced that on 10 models from three vendors, with zero variance and zero files read. Then we tested every common fix against it. Prompt directives repair it on some models and not others. A published memory architecture never beat a single line of prompt. Curated stores rot the moment maintenance slips. One intervention held on every model we ran it against, including the newest one that broke everything else: deliver the in-force decision into context, completely, and say which value wins. That is the mechanism governed memory implements.

Preliminary · internal 10 models · 3 vendors + an eleventh, released mid-study 3 fixtures · prose, graded, code isolation proven fixtures fabricated + publishable methodology & reproducibility ↗

run date 2026-07-18 to 2026-07-29 claude 2.1.197 matrix opus-4.8 · haiku-4.5 · gemini-2.5-pro/flash · gpt-5.4/mini · gpt-5.5 · gpt-5.6-sol/luna/terra after the freeze opus-5 host darwin 25.5

The failure, the popular fix, and the one that held

0/6trust the summary
ten models, 0 files read

“check the notes first” repairs

6 of 10models — and you
cannot predict which

governed delivery

6/6every model tested,
including Opus 5

We gave an agent a CLAUDE.md holding six facts about a fabricated product. All six were superseded; the current values sat in dated notes on the same disk. Asked to write a one-pager, every model wrote the fully stale version, having read zero files. Not a retrieval failure — a verification failure. The agent had the tools, the notes were trivially findable, and it never looked, because nothing in its context suggested there was anything to find. Strip the facts out of the summary and the same agent goes and finds them unprompted. The summary is the barrier.

Every mitigation below is measured against that floor, and each one fails where its responsible actor defaults: prompt directives need the model to choose to search, curated stores need a human to keep them fresh. Delivering the in-force decision into context needs neither — and on the model that made the other methods unpredictable, it was the only thing that still held.

New to Memory Bank, ADR, RAG, or spec-driven? See the methods reference →. Want the fixtures, the isolation proof, the trial counts and the two bugs our own grader had? Read the methodology →

Act I

Why the agent does not look

An agent verifies a fact only when it has a reason to doubt it. A confident, self-consistent summary supplies none — so staleness is not underweighted, it is invisible. We call this the answer-before-doubt barrier.

The floor, across three vendors and three OpenAI generations

The failure happens with all ten models of our initial matrix. The common fix does not.

Same fixture, same task, same stale summary. Under the floor arm every model wrote a fully stale one-pager and read zero source notes first. Accuracy is 0.0 with zero variance — whatever the newer generations improved, none of it touches this. The most common mitigation is a line of prompt telling the model to check the notes; it works, on some models, unpredictably.

model	floor (trust summary)	verify (“check notes”)	TierMem (published fix)	mla push (upper bound)
Claude Opus 4.8	0/6	6/6	6.0/6	6/6
Claude Haiku 4.5	0/6	0.7/6	0.7/6	6/6
Gemini 2.5 Pro	0/6	2.0/6	0.0/6	5.0/6
Gemini 2.5 Flash	0/6	0/6	0.0/6	6/6
GPT-5.4	0/6	6/6	1.0/6	6/6
GPT-5.4-mini	0/6	0/6	0.0/6	6/6
GPT-5.5	0/6	6/6	0.0/6	6/6
GPT-5.6-sol	0/6	5.7/6	0.0/6	6/6
GPT-5.6-luna	0/6	5.7/6	1.7/6	5.3/6
GPT-5.6-terra	0/6	6/6	0.0/6	6/6

The floor is 0/6 on all ten models of our initial matrix, with zero variance, including three successive OpenAI generations and the three GPT-5.6 models. The one-line directive fully repairs six of the ten models, partially improves two and leaves two at the floor — and which side a model lands on does not track a frontier-versus-cheap line. Gemini's best model ignores it at 2.0 while OpenAI's obeys at 6.0.

What the last column is measuring. The mla push arm hands the agent the fixture's in-force values, already adjudicated — retrieval, conflict resolution and selection are done before delivery rather than by the model. It is therefore an upper bound on what a delivery mechanism can achieve, and it is the right number to compare the other arms against: it is what you get when the adjudication problem is solved. Act II is where we measure what happens when it is not.

One controlled swap, everything else held fixed

The summary is the cause, not the model and not the corpus

Hold the corpus, the task and the model identical. Change one thing: the CLAUDE.md file.

SUMMARY ASSERTS THE STALE ANSWER

0 tool calls · 0 documents · ships stale

And it does this at both corpus sizes — 13 notes and 76 notes — so corpus scale is not what suppresses the search.

SUMMARY ASSERTS NOTHING

3–4 tool calls · 2–3 documents · correct

Replace the facts with “decisions are in notes/” and the same agent goes and finds them, unprompted.

The agent is not failing to retrieve. It is declining to try, because nothing in its situation suggests there is anything to find. Give it a summary that answers, and it answers. That is the whole mechanism, and every result below is a consequence of it.

Stressor 1 · a weaker model

The methods only tie when the model is strong

With Opus almost everything ties. Look at the Haiku column.

arm	Opus 4.8	Haiku 4.5
floor ↗	0/6	0/6
verify ↗ (“check notes first”)	6/6	0.7/6
RAG ↗ (all-MiniLM top-k)	6/6	0/6
Memory Bank ↗	6/6	6/6
ADR ↗ (superseded→accepted)	6/6	6/6
spec ↗ (living doc)	6/6	5.7/6
mla push ↗	6/6	6/6

On Haiku, verify and RAG collapse with 0 tool calls — the search tool worked and the answers were easy to find; the model simply did not look. Note also that 6/6 is this fixture's ceiling: it separates the methods that fail from the ones that do not, and cannot rank the ones that pass. Act III has a fixture with no ceiling.

Stressor 2 · maintenance lag

Curated notes fall behind. Raw notes do not.

The store is missing the two most recent supersessions. The raw notes still have them.

arm (under lag)	Opus 4.8	Haiku 4.5
ADR under lag	4.7/6	4.0/6
Memory Bank under lag	5.3/6	5.0/6
spec under lag	4.0/6	3.3/6
RAG (reads raw notes)	6/6	0/6 (won’t search)
mla push	6/6	6/6

Every curated-store arm degrades on both models. The agent trusts the incomplete store and does not check the complete raw notes — the same suppression, one level down. RAG reads raw notes and is immune to the lag, but that immunity is worthless on Haiku, which never searches. The push arm was never run under this stressor, so its rows here are the unstressed cells and should not be read as surviving it.

Why those two stressors and not some other grid

Sort mitigations by who is responsible, and their failures become predictions

Listing mitigations as tools — RAG, tiered memory, a decision log — predicts nothing. Sorting them by which actor is made responsible for the trusted context being correct does. Each family then fails exactly where its actor defaults, and the two stressors above are not a grid we chose: they are the two ways an actor defaults.

family	responsible actor	fails when	seen in our data
verify · RAG	the agent, on doubt	the model declines to search	✗ Haiku 0–0.7/6, zero tool calls
TierMem	the agent, on insufficiency	the summary is complete and wrong	✗ never beats one line of prompt
Memory Bank · ADR · spec	a human, at write time	maintenance lags	✗ 3.3–5.3/6 under lag, both models
CUPMem	a model, at write time	compression precedes the question	✗ capped by its own store
mla push	the system, at delivery	coverage or precedence fails	the only failure you can engineer against — Act II

Every family on this page has a failure mode, including ours. The classification's value is that it tells you which one you are buying: a method that asks the agent to act on doubt is hostage to whether that model will; a curated store is hostage to whoever maintains it; and a delivery mechanism is hostage to what it delivers and how it says it. The difference is that the last row's failure mode is the only one under your control. You cannot make a model choose to search, and you cannot make a team keep a store current. You can control what a delivery mechanism covers and how firmly it states precedence — which is exactly what Act II measures, on ourselves included.

Tested against a published system

A sufficiency router cannot fire on a summary that is complete and wrong

The obvious objection is that a real memory system would handle this, so we built the leading published one as an arm. TierMem (Feb 2026) keeps a short summary beside a complete raw log and escalates to the log only when the summary looks insufficient. We gave it every advantage: the log is complete, and the summary openly states it is a three-month-old snapshot.

model	TierMem	verify (“check the notes first”)	result
Opus 4.8	6.0/6	6.0/6	tie
Haiku 4.5	0.7/6	0.7/6	tie
Gemini 2.5 Pro	0.0/6	2.0/6	worse
Gemini 2.5 Flash	0.0/6	0.0/6	tie
GPT-5.4	1.0/6	6.0/6	worse
GPT-5.4-mini	0.0/6	0.0/6	tie
GPT-5.5	0.0/6	6.0/6	worse
GPT-5.6-sol	0.0/6	5.7/6	worse
GPT-5.6-luna	1.7/6	5.7/6	worse
GPT-5.6-terra	0.0/6	6.0/6	worse

A published two-tier architecture with a router never beat the one-line instruction, and did worse on six of the ten models we tested, better on none. Of the four ties, three models are ties because “check the notes” fails there too; the fourth is Opus, where both saturate.

The mechanism is the interesting part: escalation is all-or-nothing. Every run that escalated reached the raw log and scored 5/6 or 6/6. Every run that did not scored 0/6 or 1/6. There is no middle. On GPT-5.4 that decision came out differently across identical runs — two fixed-condition batches five days apart scored 6,0,0 and 0,0,0, so the router escalated in one trial of six. Unstable under fixed conditions is the worst thing a safety control can be.

This is the barrier restated in someone else's architecture. A sufficiency check asks “is this summary enough?” A stale summary is always enough. It answers every question. It answers them wrong.

Accuracy is not the only axis

What each correct answer costs

Averaged over 3 trials. The methods that work on Opus buy their correctness with tool calls and context, and both are billed. Cost is recorded for only some cells — every Claude CLI run and every GPT-5.5 and 5.6 run has none at all — so read this table as a within-vendor comparison, never as a study-wide ranking.

accuracy / latency / token cost · lower is better except accuracy
arm · model	acc /6	latency	work tok	context tok	cost	tool calls
mla push · Haiku	6.0	12 s	938	52k	$0.027	0
mla push · Opus	6.0	56 s	4.0k	70k	$0.26	2.3
RAG · Opus	6.0	69 s	5.0k	141k	$0.42	0
verify · Opus	6.0	64 s	4.1k	137k	$0.32	6.7
ADR-lag · Opus	4.7	119 s	8.3k	129k	$0.45	8.7
no-summary · Haiku	5.7	55 s	3.7k	517k	$0.10	15.7
verify · Haiku	0.7	11 s	0.8k	52k	$0.026	0
floor · Haiku	0.0	13 s	0.9k	52k	$0.026	0

Among the priced cells, the correct push cell on Haiku is the cheapest and fastest — $0.027, 12 s, zero tool calls — because the values are already in context and nothing has to be found. That is the mechanism's ceiling being cheap, not a product being cheap. The honest counterweight is in the same table: no-summary shows what the popular “just delete the summary” advice actually costs, at 517k context tokens to recover most of the facts on a weak model.

Act II

Why bolting retrieval onto a stale summary does not work

The obvious objection to Act I is that a retrieval layer would catch this. So we built one and measured it. Retrieving the right evidence and appending it beside the summary — what almost every memory product ships — corrected nothing. This act isolates the two properties that separate a delivery that works from one that does not, and they are the two properties governance provides.

Retrieval worked perfectly and the answer was still fully stale

Finding the right evidence is not the hard part

We started where a practitioner starts, and where most tooling stops: retrieve the relevant evidence items and append them alongside the summary. Not a mock — a real shipping render path, pointed at the identical floor fixture, with retrieval demonstrably returning the right records. On Opus 4.8 it corrected nothing. The correct values were in the context window and the model wrote the stale one-pager anyway.

Two things were wrong at once, and separating them is what makes this actionable. The block was capped below the task — it carried two of the six facts the deliverable needs — and it labelled what it carried as provisional: “may be cited as unverified… must not be presented as established fact.” The cap alone does not explain the zero, because the two facts it did deliver were not reliably adopted either. Hedged evidence loses to a confident summary.

This is our own render path, and we are naming it as ours. The provisional header above is the byte-for-byte string our Ask pipeline emits for evidence that has not been adjudicated, and the two-fact cap is what its retrieval actually surfaced on this fixture. We tested it because a benchmark that only stresses other people's tools is marketing, not measurement — and because the cell that follows is the specification for fixing it.

The stale summary is never touched in any of these runs. It stays byte-identical to the floor arm's and the evidence block is appended, deliberately, because that is what deployed retrieval does. Every cell below therefore varies only how much was delivered and how firmly it was stated, against a competing summary held constant.

Coverage × precedence, at fixed values and fixed mechanism

The specification for a correction that actually lands

Identical correct values in every row. The intact stale summary loaded in every row. What varies is the number of facts the appended block carries and the wording it uses. provisional: may be cited as unverified, must not be presented as established fact. conflict: states both values, names no winner. correction: names the asserted value, the in-force value, and which wins.

cell	facts sent	Haiku 4.5	Opus 4.8	Opus 5
provisional-2 — the shipping render path	2	1.5/6 †	0.0/6	—
provisional-6 — everything, hedged	6	6.0/6 †	0.0/6	0.0/6
conflict-6 — both values, no winner named	6	0.0/6	0.0/6	6.0/6
correction-2 — assertive, but partial	2	2.0/6	2.0/6	—
correction-6 — complete and assertive	6	6.0/6	6.0/6	5.67/6

† Scored correct, and non-compliant on the same trial. The block forbids presenting its values as established fact, and a one-pager does exactly that. Haiku reached those numbers by disregarding the instruction it was given. We report accuracy and compliance separately rather than crediting one as the other.

1. Accuracy and compliance came apart under hedging. Handed all six correct values under a provisional header, Opus 4.8 produced a fully stale one-pager: the deliverable needs established facts, the evidence was flagged unverified, so it fell back to the summary. Haiku scored 6.0 on identical input by presenting those values as established, which the block forbade. On accuracy the weaker model wins; on compliance the ordering reverses. No model produced output that was both correct and compliant under a hedge.

2. Precedence is what makes a partial append work at all. Hold the delivered facts at the same two and change only the wording from provisional to correction: both models go from unreliable to 2.0/6, the two-fact ceiling. So the cap limits how much a partial delivery can fix; it does not stop adoption. What stops adoption is declining to assert.

3. Both are required for full-task correction. Coverage under a hedge stays 0.0 (provisional-6 on Opus). Precedence carrying too few facts lands only those (correction-2 at 2.0). Only the complete, assertive block clears on both study models with zero variance.

What this specifies, and why it is a governance problem rather than a retrieval one. Both required properties are decisions a retrieval layer cannot make. Coverage of the task means knowing which decisions bear on the work in front of the agent — not which documents are similar to a query. Precedence means having already adjudicated which of two competing values is in force, so the block can say so instead of hedging. A similarity search has no basis for either: it can rank, it cannot rule. That is the gap governed memory exists to close, and this table is the acceptance test for closing it — correction-6, on every model, with zero variance. How mla implements it ↗

How much weight this can carry. Two or three trials per cell, on Haiku 4.5 and Opus 4.8, with Opus 5 added to the rows where it was run. These are existence results: they prove a correct delivered value can be ignored entirely, which is enough to falsify “delivery is sufficient.” They are not a measurement of how often that happens, and the unresolved-conflict row in particular reverses on a newer model (Act III). One cell also has an artifact gap, disclosed in the methodology.

Why an appended correction is subject to the same barrier

An unverified fragment does not displace a confident summary

This is the answer-before-doubt barrier applied one level up. The summary still answers all six facts confidently. A block that flags itself unverified de-rates its own standing while leaving the summary's authority untouched — it lowers the value of the evidence without giving the agent any reason to doubt what it already holds. A conflict notice, by contrast, supplies exactly that reason, which is why the same values behave completely differently under the two wordings.

The practical form, and it cuts against the industry's instinct. Hedging is not a safe default. It reads as caution and behaves as suppression: in our cells it produced either a fully stale deliverable or a compliance violation, and never a correct, compliant one. Every product that surfaces retrieved context as “related” or “possibly relevant” or “unverified” is choosing the losing cell in the table above. The useful question is not how carefully you can caveat a fact — it is whether you have done the work to state which value is in force, and whether you are covering what the task needs. Both are adjudication, and adjudication is the thing that has to happen before delivery, not during it.

Act III

Where the failure stops, and where it is going

A newer model, a code deliverable, someone else's test cases, and a fixture with no ceiling. Two of these cut against us.

A model released after we froze the matrix

The barrier is not permanent, and not yet gone

Claude Opus 5 shipped mid-study. Because the central claim is about what current models do, we re-ran the floor and the adoption cells on it. The floor result does not hold, and how it fails is the interesting part. Comparing a model run days earlier against one run today would confound the model with the agent CLI, which updates constantly — so we re-ran Opus 4.8 on the same build, fixture and harness, changing only the model identifier.

arm	Opus 4.8 (same-build control)	Opus 5
floor (trust the stale summary)	0.0/6 all 8 trials	4.42/6 12 trials, floored on 3
provisional-6 (hedged delivery)	0.0/6	0.0/6
conflict-6 (unresolved, no winner)	0.0/6	6.0/6
correction-6 (complete, assertive)	6.0/6	5.67/6
mla push (upper bound)	6.0/6	6.0/6

The floor becomes intermittent, not absent. Across 12 trials Opus 5 is bimodal: 3 of 12 ship a fully stale one-pager with at most one tool call, and in the other nine it explores the corpus unprompted — ls notes/, a grep over the bodies, then the specific dated notes — and recovers the facts. Nothing in the arm asks it to. The modes do not overlap on the diagnostic: floored trials made 0–1 tool calls, searching trials 4–7. And the floored trials are not a milder failure; they carry the identical stale-fact signature as Opus 4.8.

This is a weaker practical position than a deterministic floor, not a stronger one. A failure that occurs on some runs and leaves no error behind is harder to catch than one that always occurs. A team that tests once and draws a searching trial concludes the problem does not apply to them.

One of our own conclusions is overturned here, and we are retracting it rather than burying it. On the two original models, the unresolved-conflict framing corrected zero facts and we read that as “refusing to adjudicate is worse than being wrong.” On Opus 5 that reverses completely: flagged a conflict and told to verify, it verifies, reads the notes, and resolves correctly (6.0/6). We withdraw the earlier reading.

The pair is exactly what the mechanism predicts. A conflict notice supplies a reason to doubt, and on a model willing to act, that is enough. A provisional header supplies none. And note which framing still defeats the newest model completely: handed all six correct values under a hedge, Opus 5 scores 0.0/6 having read zero documents in every trial. The one framing that suppresses even this model's spontaneous search is the careful-sounding one.

Not a writing problem

The same failure when the deliverable is code that has to run

A prose one-pager invites the objection that this is a summarisation artifact. So we built the same mechanism with a code deliverable: the always-loaded file is an API quick-reference stamped “last verified 2026-01-15” asserting six obsolete constructs — create route, auth scheme, config variable, SDK method, response field, default timeout — each superseded in its own dated note among 40 fillers. The task is to write a client that creates a contract check and polls it. Grading matches exact code tokens, so there is no paraphrase to adjudicate.

arm	Haiku 4.5	Opus 5
code floor — trust the quick-reference	0.00/6 0 tool calls	5.67/6 7–13 calls
code verify — “check the notes first”	4.00/6 5–7 calls	6.00/6 7–14 calls
code correction — complete and assertive	5.00/6 0 tool calls	6.00/6 0–1 calls

Haiku floors at zero with zero tool calls in every trial, emitting four to five obsolete constructs per file — POST /v1/boards, X-Marlin-Key, MARLIN_BOARD_ID, response["board_id"], timeout=5. Every one of those calls fails against the API the notes describe. This is the difference the accuracy column hides: a floor miss is a commission — the agent asserted a dead construct into a file — where a correction-arm miss is an omission.

And the adoption result carries over. Appending the correction beside the unmodified stale quick-reference lifts Haiku from 0.00 to 5.00 with zero tool calls — adoption without search — and no arm receiving an explicit correction emitted a superseded construct in any trial. Its one remaining miss is the SDK method, which the task never forces it to reference.

Someone else's test cases, not ours

We ported an external benchmark and the signature reproduced

Every number in Act I comes from a scenario we invented, with the same model family we were testing — a real weakness. So we took STALE, an outside benchmark of 400 expert-written cases where a new fact quietly invalidates an old one, and rebuilt it in our setting: the stale value in the always-loaded summary, the current value in one dated note among 40 look-alikes, their question as the task. 272 of the 400 converted cleanly through a deterministic separation gate, so the answer key is not ours. We ran the first 40 in file order on Haiku 4.5 — a deterministic slice, not a random sample.

method (Haiku 4.5, 40 cases)	gave the OUTDATED answer	gave the current one	notes read
floor (trust the stale summary)	26 of 40 (65%)	0	0.0
verify (“check the notes first”)	27 of 40 (68%)	1	0.4
mla push	0 of 40 (0%)	17	0.0

Read the first column, not the second. STALE's questions are deliberately indirect (“recommend somewhere to relax”), so a perfectly good answer often never names the value — our keyword scoring counts that as neither right nor wrong, which understates every arm and the push most of all. Of its 40 answers, 23 simply did not mention the fact and none were wrong. A system emitting nothing would also score zero stale, so that 0/40 means something only beside the 17 correct against the floor's zero.

On cases we did not write: the trusting agent gave the outdated answer roughly two-thirds of the time, with zero tool calls on every single one, and “check the notes first” was no better than not asking (68% vs 65%) on this model.

A fixture with no ceiling — and a boundary condition

When the summary looks incomplete, the strongest model abandons it

Almost everything ties at 6/6 in the Opus column of Act I. That tie is a six-question test running out of room, not the methods being equal. So we built a harder one: 20 facts hidden among 100 notes, across five trap tiers — plain recency, a newer proposal that was rejected, an exception scoped to one customer tier, a multi-step calculation, and a near-duplicate superseded draft. It ranks cleanly, and it produced the sharpest limit on our own claim.

method (20-fact fixture)	Opus 4.8	Haiku 4.5
floor (trust the stale summary)	16.0/20	0.0/20
verify (“check the notes first”)	14.5/20 (19, then 10)	0.0/20
CUPMem (adjudicated store)	14.0/20	15.5/20
no-summary (nothing to trust)	18.5/20	12.5/20
mla push	20.0/20	19.5/20

The scope condition, which cuts against us and is the more important half. On this fixture the stale-summary failure stops reproducing on the strongest model. Faced with 20 questions, Opus decides a 20-line summary obviously cannot cover them, abandons it, and bulk-reads all 100 notes in three tool calls — scoring 16/20 without being told to look. On Haiku it still fails completely (0/20).

We think that sharpens the finding rather than shrinking it. Suppression is strongest where the summary plausibly answers the whole task — which is exactly what a project overview is built to be. Combined with Opus 5, the barrier has two axes: it needs a plausibly-covering summary and a model disposed to trust one.

Two more things this fixture shows that a 6/6 ceiling hid. “Check the notes first” is far less stable than a tidy 6.0 suggests: on identical Opus inputs it scored 19/20 on one run and 10/20 on the next, the bad run reading some notes and then taking the newest one every time. And the weaker model can find facts it cannot reason over — given the notes with no summary at all, Haiku still scored 0–1 of 4 on the multi-step trap, retrieving the right notes and failing to work out “three times the previous value.” The two fixtures measure different constructs — whether the agent looks, versus whether it can reason over what it finds — and their numbers are never mixed. This one ran on 2 models.

The strongest rival, and where it got there first

Someone else published the write-time adjudication idea before we did

TierMem is not the rival worth worrying about. CUPMem is — and it published the core idea this project had been describing as its differentiator, first, with a working prototype and a number. It appears in the same paper as the STALE benchmark (May 2026). When new information arrives it labels every older fact keep / stale / replace / unknown, propagates that judgement to related facts, then constrains what the agent may read. That is close enough to the governance model here that we are not going to claim the idea.

what is being measured (CUPMem's own benchmark)	CUPMem scores
Spotting that a stored fact is now outdated	89–91%
Refusing a question built on an outdated premise	75–78%
Actually applying the updated fact in later work	32–43%
Overall on their benchmark (best model without it: 55.2%)	68.0%

Detection is close to solved. Application is not, and application is the whole job — which is Act II restated by an independent team on a different benchmark. CUPMem spots an outdated fact about nine times in ten and applies the corrected fact downstream only a third to a half of the time.

Rebuilt as an arm on the 20-fact fixture above, it is a double edge: it rescues the weak model outright, 0.0 to 15.5, the largest improvement any arm other than the push produced on Haiku. But on the strong model it lands below doing nothing — 14.0 against a 16.0 floor — because it restricts Opus to a pre-built store that is itself only 13/20, so the safety mechanism becomes a ceiling and blocks the recovery that reading the raw notes would have given. It also has no label for “this applies to one customer tier only,” and scored 0 of 4 on that trap while scoring a clean 4 of 4 on the rejected-proposal trap we expected to be the hard one.

Do not compare these numbers. The percentages above are CUPMem's, on their benchmark — personal-assistant memory in chat, where the evidence sits in a conversation the model can already see, not in files it must choose to open. Their 68.0% stands unchallenged and our 14.0/15.5 is not a refutation of it: that is their approach rebuilt under our conditions, one adjudication pass by a small model on a product repository. No /6 or /20 score on this page should be read against any percentage in this section.

Negative results

Things we set out to prove and could not

Published at the same weight as the results that worked, because a lab that only publishes its confirmations is not measuring anything.

We tested our own feature's premise on a real codebase

The premise was false, so we stopped and built no graph arm

The idea we had not yet earned: in a big enough archive ordinary search truncates — it returns the top handful and quietly drops a related record two hops away — so a structured graph of “this replaced that” links should win. We tested it on all 223 of Wagtail's versioned release notes, 2014 to 2026, pinned to an exact commit: 1,538 searchable chunks, of which a top-10 request sees 0.65%. Then 15 hand-checked deprecation chains where answering needs two documents — the release that removed a feature, and the release one to two years earlier that deprecated it. We also made our own opponent stronger than planned: the obvious baseline was the lab's similarity search, which is weak at exact identifiers, so the baseline became BM25, which is very good at them.

chains fully retrieved, out of 15	similarity	BM25	hybrid	recency-blend
asking by exact name, top 10	8	13	13	12
asking by exact name, top 30	8	15 of 15	15 of 15	15 of 15
asking in plain English, top 10	7	5	7	6
asking in plain English, top 50	10	10	13	14

Plain keyword search answered every question — 15 of 15, with 27 of the 30 required documents in the top 5 and the worst-placed at rank 29 of 1,538. Nothing was out of reach, so there was no buried record only a graph could find. We stopped and spent no money on models.

The lesson corrects how we had been thinking: what makes search truncate is not how big the archive is, it is whether the question and the document share words. A unique identifier behaves like a hash key — grow the archive to a million documents and it still comes back first. “Bury it in thousands of notes” was the wrong mental model, and it was ours.

Then we checked we had not built a test where no effect was possible, by re-asking all 15 questions the way a developer who does not already know the identifier would. Everything inverts: keyword search collapses from 13 to 5, and at top-10 the best method finds 7 of 15. So a real gap does exist, just not the one we claimed. We also deleted a tidier structural finding of our own — that similarity and recency each find one hop and miss the other — after building the steelman version of the recency method we had just criticised and watching it refute us.

A result against the thing we most wanted to confirm

Real projects' context files mostly do not go stale — which narrows our claim

Our whole premise is a context file that has fallen behind. That deserved testing outside our own fixture, so we took the 12 real repositories from the AGENTS.md study ↗ — 10 of which ship a maintainer-written context file — and mechanically checked every file path those documents mention against the repository as it stands. A path is a claim you can verify without opinion.

result	value
Paths referenced across 7 real context files	97
Paths that no longer exist	2 (2.1%)

Hand-written context files are well maintained on the one dimension that can be checked mechanically. Two qualifications, in both directions. File paths are the easiest thing to keep current because they break loudly, while prose claims (“the default is X”) rot silently and this check cannot see them — so 2.1% is a floor on a best case, not an estimate of overall staleness. And the more useful finding came from reading the files rather than counting them: real hand-written context files are overwhelmingly instructions (“run the checks before you finish”), not claims about what is true. There is very little in them that can go stale the way our fixture does.

So we are narrowing our own claim. This failure applies to context that asserts facts — machine-generated repository summaries, memory files, auto-summaries, and our own push payload — and not to hand-written contributor guides. That also explains the AGENTS.md study's own unexplained split: generated summaries assert facts and made things slightly worse; hand-written guides assert practices and helped slightly.

Where this sits in the literature

Three papers, and the step this one measures that they do not

This area got crowded fast in 2026. An honest map, including where others got there first.

paper	what that paper found	how this study differs
STALE ↗ May 2026	400 expert-written cases where a new fact quietly invalidates an old one. The best model scores 55%. Critically, the updated evidence was already in front of the model in 77.5% of failures and it still answered from the old one.	STALE measures “found it, then ignored it.” Act I measures one step earlier — the agent never looks at all, zero files read — and Act II measures the same ignoring, isolated to the two variables that cause it. The failures stack.
CUPMem ↗ the fix in that same paper	Labels each old fact keep / stale / replace / unknown when new information arrives, then blocks the stale ones at read time. Scores 68% on their benchmark.	CUPMem published the write-time adjudication idea first, and that deserves saying plainly. The difference that survives: CUPMem governs what the agent reads, and a rule about reading cannot fire when nothing is read. Detection is near-solved (89–91%); application is not (32–43%), which is the same gap Act II reports from the other side.
Plans Don't Persist ↗ Snowflake, June 2026	Probes the model's hidden state during long tasks and finds an agent's working information lives in the text of the context window, not in the model. Once that text is dropped or compressed, it is gone. Their phrase: “context management is load-bearing.”	Two methods, one lesson. They probe what the model holds; this study measures what it does. Their honest caution is worth repeating here: re-showing old context does not automatically fix anything — which is precisely what Act II measures the limits of.
Evaluating AGENTS.md ↗ Feb 2026	Across 438 real coding tasks, adding a project context file did not improve success (−0.5% to −2% for generated ones) and raised cost 20%. The authors could not explain why.	This study offers a candidate mechanism: a file that asserts facts switches off the checking that would have caught those facts changing. It predicts their split too — files describing practices helped slightly (+4%), generated fact summaries did not.

Put together: prior work shows agents ignore evidence they have already retrieved, and that context files quietly fail to pay off. The contribution here is the step before both — a confident summary stops the lookup from ever happening — plus the finding that closing that gap requires more than delivery. The adjudication idea is not claimed here; it belongs to CUPMem.

How to read these results — and what they do not show

The push rows are a mechanism ceiling, not a product benchmark. That arm does not run mla. It is a fixed block of oracle-selected current facts placed in the prompt, standing in for what a delivery mechanism would supply. Act II is the same delivery made realistic, and it scores zero. Until that arm runs the real product end to end, these rows should not be cited as evidence about the product.
“Perfect maintenance” is a fiction and we included it on purpose. Nobody updates notes perfectly. The perfect-maintenance panel shows only that the store format is not the variable; the lag panel shows what happens in a real project.
The push arm was never run under the maintenance-lag stressor. There is no lag cell for it. Any argument that it survives that stressor is structural reasoning, not a measured result, and we are not presenting it as one.
The adoption cells carry two or three trials each, on Haiku 4.5, Opus 4.8 and Opus 5. They are existence results — they prove a delivered correct value can be ignored, not how often. One cell has an artifact gap, disclosed in the methodology.
Every closed commercial model. The failure and the mitigations held across Anthropic, Google and OpenAI, but we have tested no open-weight model, so this is not a universality proof. Gemini's numbers varied most; treat them as a trend. RAG was run only on the Claude models, with verify standing in for search-based methods elsewhere.
Correction (21 Jul 2026): we previously said the push “does not work in headless setups.” That was wrong. We had recorded that the hook injecting current facts could not fire under automation. It fires fine — confirmed with a phrase that existed nowhere except the hook's output, which the model repeated back. What actually happened is that our own harness deliberately strips that hook for isolation, and we wrote an experimental choice down as a platform limit. It went unchallenged for the length of the study.
Three of our earlier benchmarks were retracted as tautological — the graded answer lived only in a channel one arm held. A gate now refuses any benchmark whose answer is not recoverable from every arm's corpus. Details in the methodology.
No significance testing. Means and per-trial vectors only. The floor claim survives that because its variance is zero. Nothing else here should be read as a rate.

Before citing anything on this page

The phenomenon is the reproducible part. The product claim is not.

The barrier, and the ranking of mitigations against it, rest on fabricated fixtures anyone can rebuild — github.com/Meetless/stale-context-bench, MIT, every runner and grader included. The rows labelled mla push measure an idea, not a shipping integration.

git clone https://github.com/Meetless/stale-context-bench
cd stale-context-bench
node validate-isolation.mjs --model claude-opus-4-8   # prove the clean room
node matrix.mjs --trials 3 --conc 3                   # every arm x {opus, haiku}
node run.mjs --arm mf-fr-correction6 --model claude-haiku-4-5   # an Act II adoption cell

Fixtures, isolation proof, trial counts, oracle bugs and threats to validity: read the methodology →

6-fact prose (11 arms · 3 trials, 10 models) + adoption study (5 cells) + 20-fact graded (2 models) + 6-construct code + 272-case external port · isolation-proven · fabricated fixtures — no internal data