Retrieval recall — Letapis benchmarks

Why recall and not accuracy

retrieval, measured on its own

Most memory systems report answer accuracy: a language model reads what was retrieved and a second model judges the answer. That number changes with the reading model, the judge and the prompt. None of the three is retrieval.

Recall asks a narrower question with no model in the loop: of the documents that actually contain the answer, how many came back in the top k. It can be reproduced from a corpus and a query, and it is the part a memory layer is responsible for.

LongMemEval

s_cleaned · user_only corpus

Two knobs change the result and both are stated: how many documents the engine is asked for, and whether the cross-encoder reranker reorders them. Autocut, which trims the tail by the shape of the score curve, is off in every row, because the metric counts a fixed top-k and a variable-length answer measures the trimmer instead of the search.

Recall at k = 5, autocut off, one revision, one corpus
Asked for	Reranker	Recall-any@5	Recall-all@5	nDCG@5
5	on	0.9809	0.8807	0.9057
5	off	0.9666	0.9165	0.9195
10	on	0.9714	0.8329	0.8671
10	off	0.9737	0.9212	0.9255
50	on	0.9379	0.7661	0.8238
50	off	0.9761	0.9212	0.9263

Recall-any: the question counts as answered if at least one gold document is in the top 5. Recall-all: the share of gold documents that made it; questions whose answer is spread over several sessions score lower here. nDCG weighs position: first place counts for more than fifth.

Depth by depth, without the reranker

Asked for 10, autocut off, reranker off
k	Recall-any@k	Recall-all@k	nDCG@k
1	0.8735	0.2411	0.8735
3	0.9618	0.8329	0.9120
5	0.9737	0.9212	0.9255
10	0.9905	0.9666	0.9349

k = 30 and k = 50 are left out on purpose: a question's corpus averages 47.5 documents, so at those depths the whole haystack fits inside the cutoff and recall is 1.0 no matter what the search did.

split: cleaned_longmemeval_s_cleaned.json
corpus: 23 854 files · 43 347 chunks
corpus fingerprint: 490ad6a637a5d967fa0cb73f8ae3d355 (byte order)
engine revision: 0ca73c8c2fed299ab26bd6f663d327fcc62d5e25
denominator: 419 of 500 — 30 abstention, 51 with no user target, by the dataset authors' own rules
reranker served: 419 / 419 requests in every “on” row; refusals 0
duplicate rows: 0 (0.0 %)
empty results: 0

LoCoMo

locomo10 · ten multi-session conversations

A second dataset, built differently: long conversations between two people, nineteen to thirty-two sessions each, where the answer was established many sessions before the question is asked.

A hit is counted the way the dataset defines it. Each question carries anevidence field naming the sessions that establish the answer; the question counts as answered when one of those sessions comes back. The textual rule below asks a different question: is the answer string present in the returned text. It is printed alongside. It is a rougher rule: it misses whenever the conversation says "last summer" and the gold answer reads "7 May 2023".

Recall at k = 10, reranker as in production
Rule	Recall@10	Hits	Denominator
by evidence session	0.8460	1291	1526
by answer text	0.5795	889	1534

The two denominators differ on purpose: eight questions carry no parseable evidence and drop out of the first rule, so a share computed under one rule and read against the other's denominator is wrong. The evidence parser was checked against the whole dataset. It gives exactly 1 978 judgeable questions, the number the reference implementation reports.

446 adversarial questions are outside this denominator

Category 5 asks the system to refuse, and its gold answer is an empty string. Our question filter, copied from the reference implementation, drops those, which is why the denominator here is 1 526 and not the full set. They can be judged by evidence like any other question, and other published numbers include them. Changing the filter would make the denominator incomparable with every published number, so it stays.

How the text is cut

what a chunk is

Retrieval quality is decided before any query arrives, when the text is divided into chunks. A chunk holding half a thought retrieves badly no matter how good the ranking is.

Chunks cover the whole file

Each chunk starts where the previous one ended. The first starts at zero, the last runs to the end of the file, and concatenating them reproduces the file exactly. So no text can go missing.

Where a cut goes

The parser for a language knows where its structures end: a function, a class, a section. Those are the candidate cut points. The cutter takes the last candidate that still fits the budget, so it cuts between two functions before it cuts inside one.

When no proposal fits, the oversized structure is opened one level deeper and the search runs again over its parts. If nothing can be opened, the edge is placed by size, then pulled back to the nearest paragraph, sentence or word.

A chunk has to contain words

The cutter does not allow a boundary that would leave a span without a single word of content. A chunk of that kind, holding front matter, a licence header or a closing brace, looks nearly identical in every document of a corpus, so it matches every question equally well and pushes the real answer down the list.

Nothing is dropped when the boundary moves. The label ends up in the same chunk as the first words after it, and the file is still covered end to end.

The rule reads the text itself, not a list of node names. A language declares which of its elements are labels and not content; those are set aside, and the remainder is checked for one thing: is there a word here. The same check runs when the edge is placed by size.

Two size limits

The budget counts non-whitespace characters and governs where the boundary falls, so a run of blank lines cannot eat the space a chunk has for content. The capacity counts every character and is the embedder's hard limit.

Capacity belongs to the path, not to the engine. What reaches the model is the chunk plus a short header naming the file, so the room left for content differs from file to file and is calculated for each one.

Each chunk knows which cutter made it

Every chunk stores the handler and version that produced it. The version is raised by hand, when the cut has actually changed, and then only that handler's share of the corpus is read again. Rebuilding or updating the engine does not re-cut anything.

What these numbers do not say

the limits of this page

One kind of document. Both datasets are conversational sessions. Retrieval over long technical documents is not measured here at all.
No episodic memory in the corpus. Both datasets index with episodes off. A working corpus that also holds a memory layer behaves differently: there the reranker keeps episodes away from the top of the list, and that is useful.
No answer accuracy. No judge model reads anything on this page.
Why the reranker loses is not explained. It loses at every depth measured here, and loses more the deeper the request. One possible reason is that it scores a chunk and not a whole document, so a passage entirely on the topic beats a passage where the answer is one sentence in a longer exchange. I have not checked this.

What the search actually finds