The load & analysis harness

Retrieval tuning log

What was tried to improve retrieval quality, what it was measured with, and what the measurements actually supported. This page exists because the first attempt produced a confident wrong conclusion in both directions — first that a change had helped, then that it had hurt — and neither was readable from the evidence collected. The corrections are recorded here alongside the results.

The instruments, and what each one costs

Three instruments measure retrieval, and they are not interchangeable. Reaching for the expensive noisy one first was the original mistake.

What was tried

1. The cross-encoder reranker

A bge-reranker-base ONNX cross-encoder (Retrieval:CrossEncoderRerank) re-scores the top pool on the actual (query, passage) pair, which RRF fusion cannot do. Enabled 2026-08-03 on the expectation that it would fix a known precision problem.

2. A corpus fix, shipped at the same time

Chunk versioning demoted stale points by (repo, file, chunk index), so a file that shrank left orphaned tail chunks marked is_latest=true forever — and re-ingest could never clear them, because re-ingest is the path that cannot see them. RetireTailChunksAsync fixed it and contamination went 5 → 0 on the queries being tracked.

What was measured

Three paid benchmark runs over the same 24 curated questions, each question answered twice — once by an agent loop given only query_ledger, once by a loop given only grep/read over the source tree — with token counts taken from the API usage blocks and both answers graded by an LLM judge.

Read naively, B is a large regression against A, and it was first reported as one. It is not safely readable as one, for two reasons.

The control arm moved

The grep path is a control: it never touches the ledger, the index, or any retrieval setting, so nothing done on the ledger side can move it. Between A and B it moved anyway — −18.2% on input tokens and 4 → 5 on correct answers. That is model non-determinism in the agentic loop plus judge variance, and it is larger than the ledger-side difference being attributed to the reranker. Whatever the control does between two runs is the smallest difference this harness can resolve.

The headline rode a threshold sitting inside the score mass

“Correct” is a count of judge scores clearing a 0.6 pass mark. Four of the ledger’s seven failures in run B scored 0.50–0.55. Sweeping the pass mark over the same run:

The ledger count swings by 8 on where the line is drawn; the grep count barely moves. The threshold-independent figure for the same run is the mean judge score: ledger 0.690 against grep 0.198. That is the number to compare two configurations on, and it is now published beside the counts on the Context saving screen.

Run C — isolating the reranker

Run C repeated the 24 questions with the cross-encoder off and everything else held at run B’s state, making the reranker the only variable. Pairing the two runs question-by-question over the 17 that completed:

Result: the cross-encoder cost about 9% more tokens and bought nothing measurable in quality. The token effect (8.8%) is roughly twice the control drift (4.2%), so it is real. The quality effect (0.009) is an order of magnitude below the control drift (0.076), so it is unreadable — which is the honest way of saying the reranker did not help. It is now off.

Root cause: prose outranks the code it describes

The free instruments found in minutes what the paid one could not resolve in three runs. From the 44-question golden set, where every question expects a code file:

The smoke check agrees: 3 of 11 canonical queries fail, and all three fail the same way. “How are secrets loaded” returns appsettings.TeamHelp.README.md above Program.cs at 0.99 confidence — maximally confident, and wrong.

The mechanism. Both dense lanes are natural-language similarity. A document about the code matches a prose question better than the code does, because the question is prose and so is the document, while a code chunk only competes seriously in the design (intent) lane. This also explains the reranker result: a cross-encoder scores prose similarity between query and passage, so it amplifies this bias rather than correcting it.

The fix, and the guard on it

Retrieval:DocPrior scales the fused score of prose documents — .md, .markdown, .rst, .txt, .pdf only. Structured config (.json, .yml, .html, .sql) is deliberately untouched: it is frequently the correct answer to “where is X configured”, and demoting it would trade one retrieval error for another. Default 1.0 is byte-identical to no prior.

The sweep — 2026-08-04

Every value measured against the golden set and, at the same value, against four control queries whose correct answer is a document. The control column is the position of that expected document; a dash means it fell out of the top 5 and document retrieval is broken at that setting.

0.6 ships. It is the maximum on both recall@5 and MRR, and every control document is still retrieved. Below 0.5 the trade turns: code recall@1 falls and documents start dropping out, so the prior is buying nothing on either side. At 0.3 document retrieval is destroyed outright — which is exactly the failure the control queries exist to catch, and the reason the golden set was not allowed to pick the value alone.

What the prior did not fix

The smoke check still fails 3 of 11, but the failures changed class. “How are secrets loaded” no longer returns a README — it now returns Deploy-Agent.ps1. The other two return admin.html and features.html. HTML is deliberately outside the prior, so these are a different defect, not an unfixed one: a page that mentions a concept outranking the code that implements it. Recorded rather than tuned away — widening the prior until the smoke light turns green would be fitting the instrument, which is the mistake this whole page documents.

The HTML correction — 2026-08-04

The prior first keyed on file extension, which does not say what a file is. wwwroot/docs/retrieval.html is prose about the ledger; wwwroot/admin.html is the ledger. Excluding all HTML left the ledger's own handbook outranking the code it documents on 7 of the 23 questions that still missed rank 1. HTML now counts as prose when a directory segment says so — docs/, doc/, or a public-* tree — so application HTML is untouched. recall@5 0.727 → 0.750, MRR 0.583 → 0.591.

Two more free levers, swept 2026-08-04

With the paid benchmark blocked on credits, the next two candidates were both measured on the free instruments alone. One shipped, one did not, and the one that did not is the more useful result.

Per-file diversity — shipped at 1

The index holds one point per chunk, but an agent sees files. A large multi-chunk file can therefore occupy several of the five slots anyone ever reads, with near-identical content in each. On the golden set, 30 of 220 top-5 slots (14%) were a repeat of a file already shown, across 18 of the 44 questions — one query returned the same SQL migration at ranks 4, 6, 9, 10, 13 and 17, crowding out six distinct candidates.

Retrieval:MaxChunksPerFile caps chunks per file after fusion, the priors and any rerank, then backfills the deferred chunks if the cap would leave fewer results than asked for — so it can only reorder, never truncate.

A test / CI prior — swept, and NOT shipped

The same argument as the prose prior, one class over: a test names the thing it tests, repeatedly, so it matches a question about that thing better than the implementation does. After the prose prior shipped, tests still held 17 of 220 top-5 slots and 3 rank-1 positions, CI workflows 2 more — while no golden answer is a test file. The worst case was LedgerIdCarryForwardTests.cs, a file whose entire body is a comment explaining that the test is skipped, ranking 1 for "how does the ledger fuse the code and design vectors".

It was swept exactly like the doc prior, with control queries whose correct answer is a test or a workflow.

It does not ship. recall@1 and recall@5 are identical at every value — the tests being demoted were never standing between a question and its answer. The only movement is MRR +0.006, which is what shuffling already-correct results looks like. Against that: the mildest value tested already pushed a legitimate test file out of the top 5, and by 0.5 two of the three test controls were gone. There is a second cost, visible in the doc column — demoting tests promotes documents back into the freed slots, 19 → 21, partly undoing the prior that did work.

Instrument defects found along the way

Every one of these reported success while measuring nothing. They are listed because the pattern — not any individual bug — is the finding.

The ceiling this tuning is now hitting

Four ranking interventions have been measured. recall@1 has been 0.500 for every one of them. Only recall@5 and MRR ever move. That is the signature of a problem ranking cannot reach, and reading the index directly says what it is: the C# chunker emits one chunk per top-level type, truncated at 6,000 characters, so a large file is represented by its usings, fields and constructor and nothing else.

Six of the eleven remaining golden-set misses expect one of these files. The text that would answer them — HybridSearchAsync, the RRF fusion, the doc prior itself, the guarded rebuild endpoint — is not in the index at all, so no prior, cap, reranker or embedder can retrieve it. Program.cs is worse than truncated: its 4,700 lines of top-level statements are dropped entirely, because the fallback that indexes top-level statements only fires when a file has no type declarations, and it has two helper classes at the bottom.

Where this stands

• Cross-encoder: off. Measured cost ~9% of tokens, measured benefit none.
• Doc prior: live at 0.6, with HTML decided by location. recall@5 0.614 → 0.727 → 0.750, MRR 0.545 → 0.591.
• Per-file diversity cap: live at 1. Duplicate top-5 slots 30 → 0, MRR 0.592 → 0.599, controls intact.
• Test / CI prior: swept and rejected — then REVERSED on 2026-08-05. No recall gain at any value on the class-level corpus, so it stayed at 1.0. After method-level chunking it is live at 0.6 and worth +4.5 points recall@1. See the correction below: the answer changed because the corpus did.
• recall@1 did not move from any ranking change — correctly diagnosed as a chunk-granularity problem, not a ranking one. The re-ingest that followed is recorded below.
• Three smoke failures remain, now a different defect — HTML pages outranking code, which the prior deliberately does not touch.
• Run C: incomplete at 17 of 24 — the Anthropic account ran out of credits mid-run. The 24-question figure for the encoder-off configuration does not exist yet.
• Resolved 2026-08-05: two full 21-of-21 runs have since completed with zero skips, and the aggregate is safe to quote again — see Measured value. A run with any skipped question is now recorded as FAILED rather than reported as a result.

The re-ingest, and the tuning result that reversed — 2026-08-05

The granularity ceiling named above was tested. The corpus was rebuilt with method-level chunking over two runs costing $14.03, taking it from 12,961 to 23,594 chunks, enrichment coverage to 100%, and zero analysis failures in 23,594.

The first run made retrieval worse, and the per-question diff explained it in the direction opposite to the obvious guess. Splitting helps types that are genuinely huge — McpService.cs (4,840 lines) went rank 2 to rank 1 on two questions, because its old single chunk was truncated at 6,000 characters and most of the file was never indexed. It hurts types just over the threshold: AgentsMdGenerator.cs (177 lines) fell out of the top 10 entirely. Every regression sat in the 143-880 line band, where a type used to be one coherent chunk matching many query terms and was now shattered into fragments matching weakly.

Two fixes recovered it, and the second was the larger. Keeping a whole-type chunk alongside the header and members restored recall@5 exactly — but left recall@1 untouched. Five of the six lost top-1 hits sat at rank 2, beaten by near-miss siblings: ChunkerTests.cs over IngestionEngine.cs, QueryStatsService.cs over AgentQueryLogger.cs.

What the re-ingest did not buy

• Parity, not improvement. Still one question below the pre-split recall@1. The deliverable is coverage — every file now reachable at method granularity — not measured recall.
• The test prior was tuned ON the 44-question golden set, which makes the eval set partly a training set. The 0.6-0.1 plateau is flat and the mechanism is principled, but a two-question gain fitted on the eval set is not independently validated.
• The golden set structurally cannot see the benefit. It measures file-level known-item retrieval, so it cannot reward returning the right 50 lines instead of a 6,000-character truncated blob. Only the paid two-path bench can — see Measured value.

The train/test split, and two priors withdrawn — 2026-08-05

The tuning set had become a training set: TestPrior was swept on the same 44 questions used to report the result. Worse, none of those 44 had a test file as its right answer, while the prior demotes 2,210 chunks across 319 files — 9.3% of the corpus. A sweep against a set that cannot detect over-demotion can only ever look good. That is why it read flat from 0.6 down to 0.1, and that flatness was blindness, not robustness.

16 held-out questions were added — written from the source tree, never from ledger output — leading with the controls the tuning set lacked: eight whose right answer IS a test file, five whose right answer IS a document. The two halves then move in opposite directions:

Live with both priors off, over the full 60-question set:

The held-out half scores markedly higher than the tuning half. That is expected and not a contradiction: the tuning set was assembled from questions retrieval was known to struggle with, so it is adversarial by construction, while the held-out set was written to be answerable and to exercise the priors. The two halves are not comparable to each other — only each half to itself across configurations.

Open Brain — the MOTMaster Architectural Ledger · retrieval tuning log, measurements 2026-08-02 → 2026-08-05