The load & analysis harness

The end-to-end load process

One orchestrator turns eight source repositories on disk into a hybrid semantic + graph index inside Qdrant. This page walks the whole path — how a run is triggered, how only changed files are found, how each file is chunked, analysed, embedded and stored, and how the graph and corpus-wide enrichment layers are built on top — with every stage cited to the code so you can verify it.

Ingest is operator-driven; query is always-on

The heavy, occasionally-expensive work — reading source, calling the LLM, embedding, building the graph — is driven from a deploy step, while the always-on Hetzner service mostly serves fast queries over the index that ingest produced. The service is a .NET 10 minimal-API app and IngestionEngine.RunAsync (IngestionEngine.cs:109) is the single orchestrator, guarded by a process-wide SemaphoreSlim(1,1) (IngestionEngine.cs:50,111) so only one ingest ever runs at a time. A run is kicked by the deploy script over a webhook after it rsyncs fresh source; the repos live at sources/current/motmaster-* (appsettings.json:147-156).

Entry points into a run

Three endpoints start ingestion, each with a different trust model.

Loopback ingest — POST /api/ingest

Fires an ingest for all repos or one, from the admin UI or loopback. Program.cs:659 returns 409 Conflict if engine.IsRunning, validates the optional repo filter, then runs on a background Task.Run and returns 202 Accepted immediately (Program.cs:661-676). It is non-blocking: the caller polls /api/stats for progress.

Cross-machine webhook — POST /api/admin/auto-ingest

The deploy script triggers ingest on the remote host after shipping source. It is deliberately not loopback-only; the sole gate is a shared secret in AutoIngest:Secret compared in constant time (Program.cs:689-758, ConstantTimeEquals at :714). It fails closed: 503 when no secret is configured (:694), 403 on mismatch, 409 if already running. The secret is never logged.

Clean-slate rebuild — POST /api/admin/qdrant/rebuild

Drops the collection and re-ingests from scratch without throwing away the Groq analysis cache (same secret gate, Program.cs:774-803). It drops the collection (:810) then calls metrics.ClearFileHashesOnly() — clearing only the file-hash skip cache so every file re-processes, while the (contentHash → analysis) cache survives (:813-818). A path-remap rebuild re-embeds locally (cheap) but re-uses cached LLM analysis for byte-identical content, so it costs roughly $0 Groq. Contrast the loopback clear-collection, which wipes the analysis cache and makes every chunk cold.

How the deploy script ships source

Deploy-Hetzner.ps1 -SyncSources bundles the nine repos, uploads, reconstructs working trees, and flips an atomic current pointer. Each motmaster-* repo is committed and shipped as a git bundle (Deploy-Hetzner.ps1:286) — not a tarball — so the server reconstructs a real working tree with history, which lets GitEnrichmentService populate churn and authorship (:223-225). Bundles are tarred, uploaded once, git cloned into /opt/arch-ledger/sources/<yyyyMMdd>/, and ln -sfn <date> sources/current swaps the symlink atomically (:272-285). motmaster-ssl is never shipped (certs must not reach the index). The graphify CLI is (re)installed on the host every full deploy via uv tool install --upgrade --with openai graphifyy (:184-217).

A related but separate corpus — indexed developer session transcripts, used by the todo/learning-loop and quality-rubric features rather than by code search — is refreshed on its own schedule by SessionIngestHostedService (SessionIngest:IntervalHours, default 24) rather than by the source-code ingest path described here; a standard ledger redeploy re-runs that refresh (ingest + skill discovery) as one of its post-deploy steps.

A deploy merges new config keys into the live appsettings.json

The ledger is the one component motmaster-infra's deploy engine sets MERGE_APPSETTINGS=1 for (deploy/deploy.sh:41) — every other component's swap_service() either preserves the deploy directory's live appsettings.json untouched or overwrites it wholesale. For the ledger, the newly-published build's appsettings.json is renamed to appsettings.default.json and the live appsettings.json is preserved, then merge_appsettings() (deploy/lib.sh:307-320) folds any config key that exists in the new default but not in the live file into the live file — so a WP that adds a config key ships without an operator having to hand-edit the live settings file, while any key already set live (including one changed by hand, like Mcp:DefaultFormat above) keeps its live value.

The nine repositories on the roster

Live figures, read from GET /api/admin/config (Scanning:Repositories) and GET /api/distribution on 2026-08-10:

infra is the newest arrival — first ingested 2026-08-09T16:11:58Z. Until then the ledger carried no copy of the deploy engine at all, so every "how do we deploy X" answer had to be reconstructed from other repos' incidental mentions of it. motmaster-ssl is still deliberately never shipped: certs must not reach the searchable index.

File discovery and the Merkle incremental scan

Config drives everything: Scanning:IgnoredDirectories, SupportedExtensions, IgnoredFiles, MaxParallelism, and Repositories (IngestionEngine.cs:137-153; appsettings.json:109-157). Ignored-dir entries can be literals or globs (*_old, 202*), compiled to regex via GlobToRegex (:142-148,1397).

The scanner is a Merkle directory walk. WalkSubtree (:1263) computes a per-directory root bottom-up from cheap file-metadata lines (relname:size:mtime-ticks, FileMetaLine :1213) folded with child-directory roots (ComputeSubtreeRoot :1219). A single root comparison against the stored root (_metrics.TryGetSubtreeRoot :1305) proves an entire subtree unchanged without reading or byte-hashing any file inside it (:1305-1312). Changed or first-seen subtrees emit their files and persist the fresh root (:1314-1320). A no-op re-sync touches almost nothing; FilesScanned still counts skipped files so the figure stays comparable to a full-rescan baseline (:230-232).

Per-file processing

ProcessFileAsync (IngestionEngine.cs:335) runs per file, up to MaxParallelism in parallel (:237-267). Each file passes through six stages.

1. Per-file byte-hash skip — the file bytes are SHA-1'd (Sha1FileAsync :350,1367); a match skips the file entirely (:352-361). Hashing raw bytes (not decoded text) avoids running PdfPig on unchanged PDFs.
2. Structure-aware chunking — ChunkFile (:568) dispatches by extension to a language-aware chunker (see the Chunking page).
3. Deterministic text normalisation — NormalizeChunkText (:1358) folds CRLF→LF and strips trailing whitespace; the normalized text is SHA-1'd into chunkHash (:380-381) so a pure reformat yields the same hash and an analysis-cache hit instead of paid re-analysis.
4. LLM analysis with keyed cache + budget cap — the cache is keyed on (chunkHash, GroqService.PromptVersion) (:389). A miss calls _groq.AnalyzeAsync and caches only successful results (:407-421). A Groq:BudgetUsd cap (IsBudgetExceeded :1510; appsettings.json:62 = 25) short-circuits to a neutral placeholder that is indexed but not cached (:397-406).
5. Cheap structural enrichment (no LLM) — language-aware structural quality metrics are merged with the LLM rubric (:425-439), static log-template hashes are computed for runtime fusion (:443-449), and boolean pattern signatures are extracted (:451-455).
6. Local embedding into two vectors — EmbeddingService wraps a local 384-d embedder (EmbeddingService.cs:10-15). Each chunk gets a code_vector (code text) and a design_vector (embed of "[repo/layer] intent", prefixed so identical intents in different repos/layers do not collide) (IngestionEngine.cs:484-487).

Idempotent identity: stable point IDs and rename-surviving ledger IDs

The point ID is QdrantService.DeterministicId(repoId, filePath, chunkIndex) = MD5 of "repo|path|chunkIndex" → GUID (QdrantService.cs:230-234). The same logical chunk always maps to the same point, so a re-ingest overwrites it with no duplicates. The ledger ID carries history forward: at run start LoadLedgerIdMapAsync builds a (repoId|contentHash → ledger_id) map (IngestionEngine.cs:130); a chunk whose exact content was seen before in this repo under any path inherits the old ledger_id, otherwise it mints a fresh GUID (:457-464).

Writing to Qdrant: payload, two vectors, optional versioning

UpsertChunkAsync (QdrantService.cs:385) writes each chunk. The collection is created on demand with two named cosine vectors code_vector/design_vector at 384-d (EnsureCollectionExistsAsync :194-205). The payload carries repo/layer/pattern/intent/path/content/hash plus graph fields, quality (q_*), git (git_*), symbols/skeleton/line-coords, log-template and pattern-signature lists, ledger_id, concept_id and bitemporal stamps — with every optional field omitted when empty so legacy points round-trip byte-identically (:422-525). The full field list is on the Chunking page.

Versioning is a seam (Versioning:Enabled, default false — appsettings.json:52-53). When OFF (production default) a DeterministicId overwrite keeps one latest version per chunk with no is_latest written — byte-identical to pre-versioning search (:387-391,264-265). When ON, writes append under VersionedId(repo|file|chunkIndex|systemFrom) (:243-249); DecideVersionWrite (:259-270) is a pure guard that refreshes the current latest point in place when the incoming content hash is unchanged, and only mints a new version on a genuine content change, demoting the prior latest to is_latest=false (DemoteStalePointsAsync :610).

WP5.3a — how each chunk's upsert actually resolved, counted

Every run summary now counts how UpsertChunkAsync's content-hash + payload-fingerprint guard resolved each ingested chunk: UnchangedSkipped — content AND every payload-relevant field matched the live point, so no content/enrichment write was issued at all (only indexed_sha/ indexed_utc are refreshed via one lightweight SetPayload); PayloadUpdatedInPlace — content matched but enrichment (graph/git/quality) didn't, refreshed via SetPayload/DeletePayload with no new version minted; Versioned — content genuinely differed, or this is the chunk's first ingest, so a point was written in full. All three are logged at the end of every run and surfaced as unchangedSkipped/payloadUpdatedInPlace/ versioned on GET /api/stats's lastSync. They read always zero when Versioning:Enabled is off (the production default) — every chunk then follows the pre-WP5.3a plain-overwrite path and is counted only in ChunksIngested.

The graph layer: Graphify + merge + enrichment

Graph enrichment runs before file processing so chunks carry graph payload straight to Qdrant (IngestionEngine.cs:163-170), built by BuildGraphIndexAsync (:1549). The Dependency graphs page covers this in depth; the load-time shape is:

• Graphify, two modes. GraphifyRunner (GraphifyRunner.cs:26) either points at a pre-shipped graph.json (ResolveGraphJsonPath tries graphify-out/ then .graphify/, :67-76) or, when Graphify:RunOnIngest=true, shells to graphify <repo> update with a 10-minute kill-timeout and Groq budget pre-flight. It never throws on a missing binary or non-zero exit — graph is enrichment, not the critical path (:21-25,259-266).
• SQL pre-extraction + cross-repo merge. Graphify's tree-sitter SQL grammar yields nothing for T-SQL, so SqlPreExtractor.AugmentAsync runs for every repo (regex, no Groq) to add proc/function/table/view nodes (:1571-1584). Pass 2 merges every repo with a graph.json via GraphFeatureIndex.LoadAndMerge, resolving cross-repo edges by symbol name and writing all.json + per-repo slices + a manifest for /graph2 (:1587-1614).

The ordered post-merge enrichment passes each run non-fatally (:1616-1696), and order matters — each pass consumes what the previous produced:

1. ChunkAnalysisJoin — projects the modal per-file Layer/Pattern from Qdrant onto graph nodes (ChunkAnalysisJoinService.cs:38). Runs first.
2. Reachability — forward BFS from structural + analytical roots to flag orphans and dead-ends; reads the Layer/Pattern joined in step 1 (ReachabilityService.cs:70).
3. LayerSkip tagging — marks edges jumping an architectural layer; reads node Layer (:1634).
4. UsageEdge backfill — graphify is forward-only (~0 inbound edges), so this parses C# identifier usages to materialise consumer→type caller edges. This is what gives node_neighbours-up and blast_radius any caller recall (:1637-1647).
5. SemanticLayout seed (off by default) → ForceAtlas2 layout — a vector-informed JL random-projection seed (SemanticLayoutService.cs:11-46) feeds hierarchical Barnes-Hut FA2 (ForceAtlas2LayoutService.cs:52) via useExistingAsSeed; layout runs last.
6. GraphBinaryWriter — columnar binary + tile pyramid for /graph2 (:1668-1670).

After the merge, freshly-discovered graph features are pushed onto existing chunks via RefreshGraphPayloadAsync — a SetPayload changing only graph fields, with no re-chunk, re-embed, or new version (:1679-1696; QdrantService.cs:645).

Whole-corpus post-passes (full ingest only)

After all files are processed, corpus-spanning enrichment runs only on a full ingest (:271-288) via RunPostPassesAsync (:1330), in dependency order:

1. CrossStackResolver (CrossStack/CrossStackResolver.cs:63) — one scroll extracts HTTP routes + sproc refs, joins across repos, and writes back endpoint_routes/_neighbours, sproc_calls/_neighbours, runtime-fusion emitted_from, and DI implements pairs, then materialises the cross-stack / covered_by / implements edges.
2. Regenerate the /graph2 binary — step 1 added edges after the graph binary was first written (:1339-1343).
3. ConceptClusterService (Concepts/ConceptClusterService.cs:30) — k-means++ over design_vectors, writes a 4-digit concept_id to every chunk, upserts centroids into motlogs_concepts.
4. AGENTS.md generation — runs last, reading cross-stack + concept output (:1348).

A single-repo sync skips all this but still regenerates that repo's AGENTS.md (:277-288). A scheduled FullEnrichmentHostedService (Operational/FullEnrichmentHostedService.cs) periodically re-runs the post-passes so coverage does not erode, and POST /api/admin/post-passes (Program.cs:2043), /api/admin/cross-stack/run (:2021) and /api/admin/concepts/run (:1471) trigger them on demand.

Versioning bloat and garbage collection

When Versioning:Enabled, superseded (is_latest=false) versions accrue. VersionGcHostedService (Services/VersionGcHostedService.cs:32) periodically calls PurgeStaleVersionsAsync with Versioning:RetentionDays (appsettings.json:54 = 7) on a Gc:IntervalHours loop (24h). SelectStaleVersionsToPurge (QdrantService.cs:697) is a pure function that deletes only superseded versions past the cutoff — a current is_latest=true point is never selectable. When versioning is OFF (production default) the service starts, logs dormant, and returns (VersionGcHostedService.cs:20-24).

Stale-file purge — removing chunks for files no longer on disk

A per-repo directory scan (#5 above) knows every file that exists on disk right now, pending and Merkle-skipped alike (currentFilesByRepo). After a repo's files are processed, that set is diffed against Qdrant: QdrantService.PurgeStaleFilesAsync (QdrantService.cs:982-1020) scrolls every point for the repo (every version, not just is_latest), and deletes the points whose file_path is not in the current set, in batches of 512. It refuses to run when the scan found zero files for a repo that resolved to a real directory — a misconfigured or half-synced path is far more likely than "every file was deleted", and the explicit admin "delete by repo" action exists for a genuine repo removal.

The BM25 lexical index is rebuilt after every ingest, with a safety-net timer

The lexical lane's index (see Vectors & search) is built lazily inside the search path with a 5-minute TTL, so whichever request arrives first after the TTL lapses pays a multi-second rebuild. LexicalWarmupHostedService keeps it permanently warm — but its trigger changed after an RSS investigation. It originally rebuilt on a fixed timer shorter than the TTL; when the corpus grew 2.35× past that timer's design point (25.6k → 60,100 docs), the fixed 4-minute cadence became the problem it was meant to solve, allocating a multi-GB transient object graph every ~4m17s (~28 rebuilds/2h) that Server GC's per-heap segment budget never gave back to the OS — a real, confirmed churn source, not a managed-memory leak (full evidence chain in docs/evidence/rss-diagnosis-2026-09-12.md). The fix changes the driver: QdrantService.SignalIngestCompleted(), fired once per finished IngestionEngine.RunAsync run (any repo), now wakes the refresh loop immediately — the only time the corpus can actually have changed. Retrieval:LexicalWarmup:RefreshMinutes (code default raised 4 → 60) is now a safety-net ceiling bounding how long the index can go stale if no ingest ever fires, not the thing driving every rebuild; it is no longer clamped under the 5-minute on-demand TTL, since an index that is 40 minutes old between ingests is still byte-identical to a fresh one. The service itself defaults OFF (Retrieval:LexicalWarmup:Enabled=false in the checked-in appsettings.json) and is enabled on the live host via the runtime settings file.

The SCIP index build — another consumer of the same ingest-completed signal

PR #199 (6a25052) adds a second consumer of QdrantService.SignalIngestCompleted(): ScipIndexHostedService wakes on the same signal the lexical warmup above listens for and triggers ScipIndexService to build one SCIP index per (repo, commit) into /opt/arch-ledger/scip/<repo>/<sha>/ — background-triggered, off the query path entirely, never blocking a search or an MCP call while it runs. The build restores each discovered project individually and concurrently (rather than via one solution-wide restore) so one project's own failure never sinks the whole repo's build, then runs one scip-dotnet process per project — also concurrently — merging their output only at parse time; this index feeds the Roslyn → SCIP → text routing for find_definition/symbol_lookup described on the Vectors & search page.

The build is gated to default on (Precise:Scip:Enabled=true) only because the measured build time for the real host came in under a 2-minute threshold. Measured live on the host after deploy, for motmaster-agent: agent@022de380f47d50e71bf9f6f1300db7f18a75844f (csharp): 52139 definitions, 91463ms, 39424404 bytes, peakRss=790568960 — 38 MB on disk, 91.5 s wall-clock under real production load. The builder's own isolated development measurement for the same repo was faster — 43.04 s / 37.6 MB — so production ran roughly 2× slower than the number the default was set against, though still comfortably inside the 2-minute gate.

Source snapshots and graph backups

Each dated sources/<yyyyMMdd>/ directory (via the current symlink) is a self-contained backup: source working trees, per-repo graphify outputs inside <repo>/.graphify/, and a _graph-build/ directory into which BackupGraphArtefacts copies all.json, per-repo slices, manifest.json, and a self-describing info.json with build timestamp and node/edge counts (IngestionEngine.cs:1730-1782). SourceSnapshotsService (SourceSnapshotsService.cs:15) lists them for the admin panel; deleting a dated directory removes source + graph atomically.

Open Brain — the MOTMaster Architectural Ledger · generated from source 2026-07-07