When the memory ran out, every fact got a lifespan
The ceiling nobody warns you about
Section titled “The ceiling nobody warns you about”An agent starts every session knowing nothing. What it should know has to be either loaded up front — a fixed, small budget shared with the work itself — or looked up while working. Get that split wrong and the failure is silent.
Ours went wrong three ways.
It overflowed with no warning. The runtime loads the memory file whole and stops at 200 lines. We reached 28.5 KB and roughly 153 entries. Everything past the cut simply did not arrive, and nothing said so. Trimming words does not help — at one line per fact, the ceiling is about 195 facts however terse each one is.
It mixed two things that behave nothing alike. An audit found 84 of 154 entries were plain reference material — port numbers, identifiers, API shapes — sitting in a store meant for behaviour. Reference material grows with every service you document and is perfectly safe to look up when you need it. A hard-won lesson is the opposite: it has to be already in mind, because it fires exactly when the agent is confident it is right. You do not go looking for a warning about a bend you do not know is coming. That one sentence is the whole design.
Then it silently forked. The runtime files memory by working directory. When we moved to five concurrent sessions in sibling directories, one store became nine, holding 277 lessons, with no overlap at all — five colleagues each keeping a private notebook and never comparing them. Any one session could see about 14% of what the estate had learned, and the same mistake could be paid for five times over. Nothing flagged it, because a notebook holding a fifth of the knowledge looks exactly like one holding all of it.
Carry what you would never think to ask for. Look up the rest.
Section titled “Carry what you would never think to ask for. Look up the rest.”Nothing new was built to hold the knowledge. The estate already documents itself in version control, so the search service indexes that rather than keeping a second copy — one source of truth, two audiences: people read the website, the agent searches the index.
The carried half holds shapes, not stories. When the nine notebooks were merged, the obvious move was to carry a pointer to all 277 lessons. That just rebuilds the ceiling. Instead we hand-wrote 24 failure classes — named shapes of self-deception, such as an empty search result is not proof the thing is absent, or a test that would still pass with the feature removed. A class warns about a mistake not yet made; a pointer only labels one already paid for. The 277 stories moved to the looked-up half, each tagged with the class it belongs to.
Saving a lesson is one call, and it proves itself. Lessons get written because writing one is cheap; make it a commit-and-wait and nobody writes any. But the call does not trust its own success message — it writes the page, reads it back, indexes it, then searches for a unique marker to confirm the lesson really is findable. Anything less returns “capture failed” and says plainly that the lesson was not saved.
What is kept, what expires, and when
Section titled “What is kept, what expires, and when”This is the part most teams get wrong, and it is where a blown memory budget is actually fixed.
Deleting by age is the obvious answer and it is the wrong one. A warning about a system still in production is as valid at ninety days as at one, and the rarely-needed lesson — the once-a-year migration trap — is precisely the one worth keeping. Deleting by “least recently used” fails too, because the runtime never reports which memories it actually recalled. So nothing expires on a timer. Each kind of knowledge expires when the thing it describes ends its life.
| Kind | Example | Kept | Expires when | How |
|---|---|---|---|---|
| Rules and failure classes (carried) | “an empty search result describes your query, not the world” | Indefinitely | Never automatically | Reviewed by hand at the end of a session; the list changes about monthly |
| Reference (looked up) | how the backup server’s API is called | Indefinitely | Never — it is documentation | Superseded by editing the page, in the same change as the thing it documents |
| Work context (looked up) | “this migration is half done; these hosts remain” | While the job is open | The job it names is closed | Flagged automatically, then a seven-day grace period, then the note and its index entry are deleted by a daily sweep. Reopening the job cancels the deletion |
| Lessons (looked up) | “this deploy reports success but silently skipped the restart” | Indefinitely | Only when the thing it is about is retired | Decommissioning a service lists the lessons that name it, for a human to confirm. It never deletes on its own — the name usually lives on at a replacement |
Two details do the real work.
The grace period is a rescue window, not a delay. When a job closes, its notes are marked for deletion in seven days rather than removed outright. That mark is the prompt to ask whether anything in them was a durable lesson worth rewriting as one. Clutter is cleared as a by-product of finishing work, and the valuable residue gets one last chance to be saved.
Freshness is checked when something is used, not on a schedule. Every lesson records how strong its evidence is and when that was last confirmed. When the agent pulls up an old one and acts on it, it re-checks that lesson against the live system there and then, and re-dates it. There is no periodic staleness sweep, because re-verifying a corpus nobody is reading is maintenance on a machine nobody uses — and routine “this might be out of date” warnings only teach the reader to ignore them.
How we know it is working
Section titled “How we know it is working”The risk is not the service falling over. It is the service staying up and quietly returning nothing — every dashboard green, every answer empty.
| Signal | Fires when |
|---|---|
| Nothing indexed | The index is empty, so every search silently returns nothing |
| Rebuild failing | An index rebuild ran and threw an error |
| Index behind the documents | The documents changed and the index has not caught up |
| Meaning-based search erroring | The half of search that understands wording, rather than exact words, is failing |
That last one is there because it happened. A library update removed the function the search path called. Search quietly fell back to exact-word matching and returned nothing for any hyphenated term — while the index held 727 perfectly good entries nobody was querying. Every signal on the writing side stayed green throughout.
And every deploy proves the one thing that would otherwise fail invisibly. A check saves a lesson as one session and demands it back as a different one, using a marker nothing else could contain. Writing and reading as the same session was rejected deliberately: that test passes today, before any of this was built, and would have gone on passing through all seven weeks of the split. A test that passes when the feature is missing is not a test. If it fails, the deploy is rolled back automatically.
Where this is behind the industry
Section titled “Where this is behind the industry”The thing being measured is no longer the thing that is growing. The original guard watches the old memory file, and it works — that file now sits at a dozen lines against a ceiling of 200. But the weight moved to the operating-instruction files, now roughly 169 KB, on the order of 40,000 tokens spent before any work begins. Nothing measures it and nothing warns. The alarm is on the door that stopped being used.
Search quality is argued, not scored. A mature practice keeps a set of questions with known-correct answers and measures how often search finds them, every release. We have no such score, so a gradual decline in results would be invisible: the alarms cover empty and behind, never worse. This is the biggest gap.
Results are returned in the order they are found. Merging two kinds of search and handing back the top few is where we stop. A second pass that re-reads the shortlist and re-orders it is usually the cheapest large improvement available.
One of everything. A single service, a single index, a single host doing the language processing — proportionate for one operator, not something a team should depend on.
Published is not searchable. The website and the search index are two different artifacts and a full rebuild takes about thirteen minutes, so a page can be live and still absent from search. “It published successfully” is not evidence the agent can find it.
Next steps
Section titled “Next steps”- Put a number on what the agent carries. Measure the size of the always-loaded files, warn before they get too big, and chart it — exactly what the old memory file already has. This is the gap most likely to bite, because that growth is invisible.
- Score the search against a set of known questions. Fifty to a hundred real questions with known-correct pages, checked on every release. It turns every future change to search from an argument into a number.
- Try re-ordering the results, and let that score decide. Accepted or rejected on the measurement, not on what the literature says.
- Ask the deploy one real question. The silent failure above would have been caught in minutes by a single search with a known answer.
- Count the searches that come back empty. Today a genuine gap and a badly-worded question look identical — which is the estate’s own most-recorded mistake, in miniature.