The load & analysis harness
Chunking — splitting source into meaningful units
Before anything is analysed or embedded, each file is split into chunks — the atomic unit of everything downstream. The ledger uses one language-aware chunker per file type so a chunk is a whole class, a whole stored procedure, a whole markdown section — a single coherent unit of meaning — rather than a blind character window. This page covers every chunker, the body-folding skeletons, line coordinates, the full Qdrant payload, and the two named vectors.
Chunking dispatch — one chunker per file type
IngestionEngine.ChunkFile(filePath, content) is a pure switch on the lowercased extension
(IngestionEngine.cs:613-628): .cs → ChunkCSharp (Roslyn); .md →
ChunkMarkdown (heading hierarchy); .pdf → ChunkPdf (page grouping); .sql → ChunkSql
(object boundaries); .ts/.tsx/.js/.jsx/.vue/.svelte → ChunkTypeScript (brace-depth); everything
else → ChunkPlain (character window). Each returns a List<ChunkPiece> — a readonly record
struct with Text, Type, StartLine, EndLine, Symbols, Skeleton, all defaulting to "absent"
(:560-566). The Type (cs_class, cs_file, sql_proc,
ts_function, ts_file, md_section, pdf_pages,
plain) is persisted as chunk_type so retrieval can signal structural-vs-blind
confidence and filter blob chunks (Models.cs:37-41).
A chunk starting mid-function embeds to a diluted vector with a vague intent. Cutting on real declaration boundaries means one embedding equals one coherent unit of meaning — the single most important lever on retrieval quality.
Roslyn-aware C# chunking
ChunkCSharp (:643) parses via CSharpSyntaxTree.ParseText and selects
BaseTypeDeclarationSyntax whose parent is a namespace or compilation-unit (top-level types,
:592-595). For each type it prefixes the file's using directives and a
namespace X { } header onto the declaration so the snippet is self-contextualizing
(:609-616). Over the 6,000-char cap (CodeChunkMaxLen, aligned with Groq's prompt window
:582-585) the type is split per member — a header chunk carrying the type's
declaration, fields and doc-comment with the bodies folded away, plus one cs_member chunk per
method, plus a cs_class_full chunk holding the whole type truncated. Before 2026-08-05 an
oversized type was simply hard-truncated with // ... [truncated], which meant 62.7% of the
largest files was never indexed at all: McpService.cs (4,840 lines) kept its usings and
constructor and lost everything after them. The line
span comes from Roslyn's GetLineSpan() via LineSpanOf, 1-based inclusive against the
original file (:627-631). Symbols are projected by CollectCSharpSymbols, emitting a
ChunkSymbol{Name,Kind,Parent,StartLine,EndLine} per type + method/ctor/property
(:636-679). Files with no type declaration (top-level statements) become a single cs_file
chunk (:598-607).
This is the flagship syntax-aware unit: a query for BookingService.Cancel resolves to
the method, with exact coordinates — not three fuzzy 3KB windows, and no longer a whole class
blob whose tail was thrown away.
Keeping the whole-type chunk alongside the members is deliberate and was measured. Splitting alone regressed golden-set recall@1 from 0.6364 to 0.5682: it helps genuinely huge types, and hurts types just over the threshold, which used to be one coherent chunk matching many query terms and became fragments each matching weakly. Keeping both shapes recovered recall@5 exactly. See the retrieval tuning log.
Every per-member chunk now self-attributes its own symbol
A live session watched query_ledger serve the SAME stale point —
McpService.cs L2613-2683 — as the top hit for four unrelated symbol names
(CallBlastRadius, AppendCitedBlock, ResolveSourcePath,
policy_check). The stale point itself was ordinary re-ingest lag, but investigating why
nothing could catch the drift found a real chunker defect: CollectCSharpSymbols(scope) only
harvests a symbol when a BaseTypeDeclarationSyntax is found under scope — correct
when scope is the whole type, but the per-member split path calls it with
scope = m, a single method/ctor/property node, which is never itself a type declaration and
(normally) contains none. So every cs_member chunk in the codebase carried an EMPTY
Symbols list — a member's only symbol record lived on its type's own multi-thousand-line
header chunk, never on the member's own individually-embedded point, so nothing could ask "does this chunk's
claimed range still match a live symbol with that name?" The fix self-attributes first: when scope
is itself a method/constructor/property declaration, its own ChunkSymbol is added before the
existing type-walk runs — additive, so the type-header chunk still lists every member too.
query_ledger still
does not cross-check a served hit's stored symbol against a fresh live parse before serving it — a reasonable
next step, filed as a follow-up rather than fixed here.Without a self-symbol, a stale or orphaned per-member point had no way to be caught — it looked as
authoritative as a fresh one for every one of the ~180 methods in a large class. Every cs_member
chunk can now be checked against its own claimed identity.
What the analyser sees is not what is stored
A member chunk is stored bare — the method body with a one-line // Namespace.Type header.
The chunk sent to the analysis model carries more: the enclosing type's skeleton, so the
model writing the intent can see what the type actually does. ChunkPiece.AnalysisText holds
that second text; empty means the two are identical, which is every non-member chunk.
They are split because they are read by different consumers. Measured, folding the skeleton into the
chunk lifted design-lane recall@1 from 0.591 to 0.659 — but the chunk text is also what
query_ledger renders, capped at 600 characters, so storing it would have opened every
member hit with a type outline instead of the method body. The gain comes from the intent being better
informed, not from what is stored. The analysis cache is therefore keyed on the analysis input:
two identical method bodies in different types receive different context and must not share a cached
intent, since telling them apart is the entire point.
Skeleton / body-elision folding
A body-elided rendering — signatures, attributes and doc-comments kept, method bodies replaced with
{ … } — is stored alongside the full body. In C#, CSharpSkeleton re-parses the local
snippet and runs CSharpBodyFolder : CSharpSyntaxRewriter, replacing method/ctor/local-function/
expression bodies with { /* … */ }; a parse failure yields "" (:684-728). In TS/JS,
TsSkeleton is a line-based brace-depth fold — depth-0 signature lines kept, inner body →
{ … } (:1096-1135). Markdown, PDF and plain carry no skeleton. It is stored in the
skeleton payload key only when non-empty (QdrantService.cs:464), and
query_ledger serves it via detail:"skeleton" (McpService.cs:305).
A 300-line service is roughly 4K tokens of body but about 400 of skeleton. "What is the surface of this class?" is answered at about 10% of the token cost.
Markdown section chunking
ChunkMarkdown (:943) matches ATX headings (^#{1,6}\s+…), walks the
document maintaining an ancestor stack, and slices heading→next-heading as a section, emitting an
md_section prefixed with a <!-- Path: A > B > C --> breadcrumb
(:763-765). Over the 12,000-char cap (MarkdownChunkMaxLen) it splits on paragraph
boundaries via SplitLongSection, each part re-emitting the breadcrumb + (part n/m).
Line ranges come from LineAtOffset. A file with no headings falls back to ChunkPlain.
Breadcrumbs give prose the same "where does this belong?" context that namespace prefixes give code — a section from deep in a spec still knows its place in the hierarchy.
SQL stored-procedure — and table — chunking
ChunkSql (IngestionEngine.cs:1158) finds object starts by regex and slices
CREATE→next; the first object absorbs any
preamble. Each becomes a sql_proc chunk with one ChunkSymbol naming the
schema-qualified object, kind mapped sql_procedure/function/view/trigger. Ad-hoc migration
scripts with no object definitions fall back to ChunkPlain.
CREATE TABLE is a boundary too
The original boundary regex deliberately omitted TABLE, and consolidated schema files paid
for it. SQL/49_mot_master_agent_tables.sql is 69 CREATE TABLEs and 2 triggers, so
it matched at 2 boundaries: the first chunk spanned everything up to the first trigger and
was then cut to CodeChunkMaxLen (6,000 chars). Measured on the real corpus
(IngestionEngine.cs:1129-1144):
| File | Bytes | Chunks (before) | Indexed |
|---|---|---|---|
SQL/49_mot_master_agent_tables.sql | 176,386 | 2 | ~12 KB — 93% invisible |
SQL/0100_mot_master_wiki_schema.sql | 102,411 | 1 | ~6 KB — 94% invisible |
A table the ledger cannot see cannot be cited, so column questions fell back to prose. With
Chunking:SplitSqlTables (default on) each CREATE TABLE starts its
own chunk, carrying its own sql_table chunk type and symbol kind — so a table now resolves
through symbol_lookup and answers for its own columns. The flag is a true revert: with it off
the boundary set is the original regex and the output is byte-identical to the pre-change chunker, pinned by
SqlTableChunkingTests.FlagOff_OutputIsByteIdenticalToLegacyChunker. Re-chunking costs a paid
Groq re-ingest, which is why it is gated at all — a bad re-ingest is undone by config, not by another
re-chunk.
CoverageIngestDryRunTests): across 235
.sql files, chunks go 1,232 → 1,457 (+225), with 67 files changing and 298 chunks billable.StartLine used to land one line early whenever a blank separator line sat above a
CREATE: the boundary regex used \s* under RegexOptions.Multiline, and
\s matches the newline too, so ^ anchored on the blank line and the match walked
forward onto CREATE — landing the object's start line on the blank line, not its own first line.
Fixed (bug 07738c5f) by requiring [ \t]* instead, which still allows an
indented CREATE but cannot cross a newline, so ^ is forced onto the
CREATE line itself.ChunkSql shapes chunks; separately,
SqlPreExtractor.cs is a graph-side regex pass scanning .sql for definitions and
references (EXEC/FROM/JOIN/INSERT/…) that emits nodes and edges into graphify's graph.json — because
graphify's tree-sitter SQL grammar yields nothing for a stored-proc-first schema (:14-57). It does
not affect chunk boundaries.The MOTMaster schema is stored-proc-first, so a per-proc chunk gives each procedure its own intent, embedding and coordinates — the natural unit developers reason about.
TypeScript / JS / Vue / Svelte chunking
IngestionEngine.cs:621-625). It used to be decided by content: any file containing
the string <script had its script blocks extracted and everything else discarded. A plain
.js file that merely mentions a script tag — a loader writing
'<script src="…"></script>', a template string, a doc comment — matched, the
captured group came back empty, and the file produced zero chunks. Not truncated:
unreachable. Measured on the corpus, 5 files come back — brand-config.teamhelp.js twice in agent and twice
in wiki, plus scout's stripe-checkout.js — and a 6th, agent's embed/p.js, goes
from 50 indexed characters to its whole 4,594. A .vue whose <script> is src-only now falls back to
the raw component rather than vanishing; a genuinely empty file still yields nothing.ChunkTypeScript (IngestionEngine.cs:1270) pulls
<script>…</script> from single-file components via
ExtractScriptBlocks (ignoring template/style, :1419-1425),
splits into lines, tracks brace depth with UpdateBraceDepth (approximate — braces in strings or
comments only coarsen a boundary, never crash, :1429-1436), and marks a boundary at depth-0 lines
matching TsTopLevelDecl (function/class/interface/enum/const/type). Leading imports ride the first
chunk. Each is tagged ts_function, gets a best-effort symbol via TsSymbolFromLine, plus
a TsSkeleton.
.vue/.svelte, chunking runs over the extracted
script text whose line numbers do not map to the original file, so StartLine/EndLine
are left unstamped (0) rather than wrong; raw .ts/.js are stamped
(:999-1002,1044,1055-1056).This closes a formerly-documented gap — the blueprint and REBUILD_PLAN describe TS/JS falling to ChunkPlain; the dedicated chunker is the fix, and the UI repo is almost entirely TS/Vue. The semantic-dilution warning now applies only to types still on the plain fallback.
Tree-sitter chunking — a structural chunker for languages that had none
PR #198 (138cc08) adds a second, additive chunking path to IngestionEngine.ChunkFile,
selected per language behind config flags — Chunking:TreeSitter:<Lang> — binding
TreeSitterLanguagePack 1.10.9 (NuGet, MIT). Where a flag is on, that language's chunker walks the
real Parser/Tree/Node API and classifies node kinds
(class_declaration, method_declaration, constructor_declaration,
property_declaration, …) the same way ChunkCSharp classifies Roslyn syntax node types
— the point being languages that previously had no structural chunker at all (Razor was
ChunkPlain; TypeScript/JavaScript already had the brace-depth ChunkTypeScript above) can now be
chunked on real declaration boundaries instead of a blind window.
Defaults: Razor, JavaScript, TypeScript = true;
CSharp, Sql = false. No production flag was flipped as part of shipping this
and no re-index has been performed — every figure below is a read-only measurement over the corpus mirror.
| Language | Corpus | Files | Chunks before → after | Member symbols before → after |
|---|---|---|---|---|
| Razor | wiki .razor+.cshtml | 94 | 242 → 258 | 0 → 309 |
| Razor | motlogs/ui .razor | 111 | 384 → 214 | 0 → 25 |
| JavaScript | wiki .js/.jsx | 17 | 110 → 108 | 95 → 95 (parity) |
The plain-text chunker produces zero symbols for Razor — a blind 3,000-char window — so a Razor member
could not previously be resolved to a chunk by symbol_lookup at all; tree-sitter finds real
method_declaration/property_declaration/constructor_declaration nodes
inside @code blocks. JavaScript's parity (95 → 95) is an honestly-reported non-win on this
corpus's actual files: it is mostly free functions, not classes with multiple methods, so the per-method-chunk
advantage tree-sitter gives a class-shaped file does not show up in this sample — it is real and reproducible
on a class, just conditional on the file containing one. There are zero real .ts/
.tsx files anywhere in the indexed corpus, so TreeSitterTypeScript is proven
correct only on unit fixtures, never measured against production content.
Line coverage after the fix: 100% on every file in all three measured sets (0 of 94, 0 of 110, 0 of 17 files below 100%), matching the plain-text chunker's by-construction full coverage. This was not true on first measurement — a review before merge found the Razor and JS/TS chunkers could silently drop content (worst case 8.7% coverage on one 1,395-line Razor file, losing 1,274 lines of markup) while their chunk/symbol counts still looked like an improvement; both were fixed to split rather than truncate an oversized unit, and are now covered by regression fixtures asserting no chunk carries a truncation marker.
class_declaration wrapper
entirely on some real files (including this ledger's own 4,226-line QdrantService.cs) — flipping
it today would make some of the codebase's largest, most-queried files far less retrievable, the exact
regression per-member splitting exists to prevent. SQL: the bundled grammar is generic/ANSI,
not T-SQL — CREATE OR ALTER PROCEDURE and a trigger's ON clause both fail to parse —
and boundary agreement on 50 sampled real .sql files is 0.0%, every disagreement
the same grammar-mismatch failure mode.Razor and JavaScript now have a real structural chunker available where none existed, with a measured, unconditional win for Razor (zero symbols to real ones, no coverage regression) — and the two languages withheld (C#, SQL) are withheld because the evidence, not a guess, says the bundled grammars are not yet good enough to replace what is already in production.
ChunkPlain character-window fallback
ChunkPlain (:1094) is the last resort for any extension without a structural
handler (Python, YAML, JSON, SCSS, Razor/cshtml, csproj, dotfiles). The whole file is one plain
chunk if ≤3,000 chars; otherwise it windows at target 3000 / overlap 200 chars with best-available line ranges
and no symbols or skeleton.
chunk_type="plain"
lets retrieval down-weight blind-window chunks.It guarantees every file is indexed no matter its type, and the plain tag is an honesty signal
that downstream ranking can use to prefer structural chunks.
PDF page chunking
ExtractPdfText renders to text with \f-delimited <!-- Page N -->
markers (:1383-1387); ChunkPdf (:1026) groups pages to about 8,000 chars
and emits pdf_pages prefixed with <!-- Path: file.pdf, pages 3-5 -->. Oversize
single pages are paragraph-split; a scanned or image-only PDF still gets one placeholder chunk.
Design documents can cite the exact page that grounded an answer, so a claim traces back to a specific page of a specification rather than a whole PDF.
Line-coordinate stamping → surgical extraction
Every structural chunk carries a 1-based inclusive line range. Coordinates originate in the chunkers
(LineSpanOf for C#, LineAtOffset for SQL/MD/plain), flow through
ChunkPiece.StartLine/EndLine → CodeChunk.StartLine/EndLine (Models.cs:43-49)
→ the start_line/end_line payload keys, written only when >0
(QdrantService.cs:461-462). The read_span MCP tool (McpService.cs:3638-3672)
takes a file plus start and end, resolves the path via ResolveSourcePath, returns just those lines,
clamps to EOF, and caps at Retrieval:ReadSpanMaxLines (default 400) via the unit-tested
ResolveSpanRange. Search results render each hit's L<start>-<end>.
A 1,200-line file is about 15K tokens; the matched span is a few hundred. Coordinates turn "open the file" into "read exactly these lines".
The chunk payload — every field written to Qdrant
CodeChunk (Models.cs:6-98) is assembled in IngestionEngine
(:466-502) and written by QdrantService.UpsertChunkAsync (:422-519):
| Group | Fields |
|---|---|
| Identity / core | repo_id, file_path, chunk_index,
content, content_hash, layer, pattern,
intent, ingested_utc (intent/layer/pattern from Groq) |
| Stable IDs | ledger_id (survives renames), concept_id
(cross-language cluster), chunk_type |
| Structural | start_line, end_line, skeleton,
chunk_symbols (compact JSON short keys n/k/p/s/e, capped 64) |
Quality q_* | structural q_cyclomatic, q_cognitive, q_loc,
q_comment_density, q_max_nesting, q_param_count, q_dup_hash; LLM rubric (1-5) q_readability,
q_testability, q_error_handling, q_single_responsibility, q_naming, q_risk_note; joined
q_test_coverage, q_error_emission |
| Graph | graph_in, graph_out, graph_cluster, graph_centrality, blast_radius (only
when graphify populated the file) |
| Git | git_last_modified_utc, git_author_count, git_churn_90d, git_last_sha,
git_last_msg |
| Freshness (WP4.1a) | indexed_sha (the repo's git HEAD sha at the moment
THIS chunk was ingested), indexed_utc — backs the per-file
staleness marker and is refreshed via a lightweight SetPayload
even when the chunk's content is otherwise unchanged |
| Analysis provenance | intent_source — "groq" when the LLM
actually analysed the chunk, "none" when it was indexed with a neutral placeholder because
no Groq:ApiKey was configured or the lifetime budget cap was hit (see below) |
| Bitemporal | valid_from_utc (code era), system_from_utc (indexed),
omitted when null |
| Versioning / lifecycle | is_latest (true only when versioning enabled);
deprecated (nullable, set by MarkDeprecatedAsync; query_ledger
post-filters deprecated unless include_deprecated) |
Every optional field is omitted when empty, so pre-existing points round-trip unchanged across schema additions — the payload can grow indefinitely without a migration or a re-ingest of legacy points.
Graceful degradation when Groq is keyless or over budget
A missing Groq:ApiKey used to crash the whole app at startup, because
GroqService is a singleton dependency of IngestionEngine and its constructor threw.
BuildClient (GroqService.cs:97-116) now logs a warning once and returns a null
client instead: ingest carries on, and every chunk is indexed with a neutral placeholder analysis and
intent_source="none" (QdrantService.cs:598,625) rather than an LLM-derived intent.
The same field is written when the Groq:BudgetUsd cap is hit mid-run. Ingest.LastSync
surfaces an IntentSkipped count on /api/stats (Models.cs:383-389) so a
keyless or budget-capped run is visible on the dashboard rather than looking like a quiet, fully-analysed
ingest.
A keyless deploy — a fresh environment, a leaked/rotated key, a spend cap — degrades to a searchable but intent-poorer index instead of not booting at all, and the degradation is visible rather than silent. The efcore scale trial (Vectors & search) checked this field before trusting any recall number from that run.
Named vectors — code_vector + design_vector
The collection is created with two named vectors, both cosine at 384-d (QdrantService.cs:204-207).
At ingest CodeVector = Embed(text) and
DesignVector = Embed($"[{repoId}/{layer}] {intent}") — the intent prefixed with repo + layer so
identical intents in different repos do not collide (IngestionEngine.cs:484-487). They are searched
as separate lanes fused by hybrid search (see Vectors & search). A
QualityVector also exists for find_similar(mode:"quality").
"Find code that looks like this" (code axis) and "find code that does the same job" (design axis) are different questions, and both are answered from the same stored chunk without re-embedding.
Idempotent IDs, content-hash normalisation, versioning
The deterministic ID is MD5 over repo|file|index (QdrantService.cs:230-235) so the
same chunk overwrites in place. NormalizeChunkText folds CRLF→LF, trims trailing whitespace and
trims blank edges before hashing and embedding, so a cosmetic reformat yields the same content_hash
and a Groq cache hit (IngestionEngine.cs:377-381,1358-1365). The ledger_id is carried
forward when the same repo|contentHash was seen before under any path, else a fresh GUID. In the
versioned write path, DecideVersionWrite reuses the current latest id and refreshes it in place on
an unchanged hash (no new version), and only mints a VersionedId (MD5 over
repo|file|index|systemTicks) on a changed hash, demoting the prior latest.
The in-place-on-unchanged rule is what stops graph-payload-only re-upserts from re-versioning the whole collection every cycle — the same fix that killed version bloat, expressed at the chunk level.
Structural drift over chunk symbols
StructuralDrift.Compute (StructuralDrift.cs:53-98) matches SymbolSnapshots by
kind-qualified name: only-in-after → Added, only-in-before → Removed, same name different signature →
SignatureChanged, same signature different start line → Moved (a moveThreshold suppresses trivial shifts). It is
pure and tested, and consumes the same Symbols projection the chunkers stamp — powering the
drift_since tool (see Context saving).
The delta of "what changed structurally" is a few hundred tokens versus tens of thousands for two full-file dumps.
TOON — dense serialisation of chunk results
query_ledger can format hits via FormatResultsToon
(McpService.cs:1841,4261): a single legend
repo|layer|pattern|path|Lstart-Lend⟫snippet then one row per hit.
Lstart-Lend is fed straight from the stamped coordinates so every TOON row is actionable by
read_span. It is a serialisation boundary only — retrieval and ranking are unchanged, and it is
A/B-testable.
The wire format is itself a token cost paid per hit; the row-oriented DSL strips repeated keys and fences, so the agent spends context on snippet content, not scaffolding.
ChunkTypeScript has existed
since the brace-depth chunker above; Razor now has a structural tree-sitter chunker too, on by
default (see "Tree-sitter chunking" above). The plain-fallback caveat applies only to the
remaining unsupported types (Python, YAML, JSON, SCSS, csproj, dotfiles).Per-member chunking's cost is concentrated, not uniform
The efcore scale trial (Vectors & search) measured what per-member
splitting costs at scale: because it multiplies Groq calls by member count, ingest wall-clock is dominated by
a minority of files rather than spread evenly. Slowest single files: Snapshot.cs 1,059s,
EFCore.baseline.json 921s, CoreStrings.Designer.cs 892s — 15-18 minutes each.
Throughput swung between ~156 files/10min in ordinary source and ~13/10min inside
test/EFCore.SqlServer.FunctionalTests/Query, where a single file holds hundreds of test methods;
efcore contains 17 *.Designer.cs generated files, 33 .cs files over 100KB and 96
over 50KB. The corpus was deliberately not narrowed for the trial (21 of 260 change-task reference paths fall
in those trees), so the finding is a size/member-count guard to add before the next scale run, not a call to
exclude generated or test-dense files from the index.
A single generated file or a functional-test suite with hundreds of methods can dominate an ingest's wall-clock and Groq spend out of proportion to how often anyone actually queries it — worth guarding before pointing this chunker at a much larger foreign codebase.
Open Brain — the MOTMaster Architectural Ledger · generated from source 2026-07-07