Pipeline anatomy — the through-line¶

One folder of messy files becomes a queryable corpus and a knowledge graph, and a query that no single retriever can answer gets answered. This page is the spine: the whole system, end to end, in one read. Each stage gets a sentence or two here and a chapter of its own underneath.

Everything below is the stack as it is actually configured today — the five pipeline files the benchmark campaign ran, at benchmark/sota-campaign/stages/s3-pipeline/pipelines/. Model ids, defaults and config values are read out of the implementation, not out of prose: where a doc and the code disagreed, the code won and the doc was corrected.

The through-line, in one paragraph¶

A folder of files is enumerated and each file is content-addressed and stamped with provenance. Anything oversized, spoofed or zip-bombed is quarantined before it is parsed. What survives is turned into markdown — by a plain byte decoder if it is born-digital, by a served vision-language model if it is a scan or a page image — and split into overlapping, offset-preserving chunks that can still name the exact page and character range they came from. Each chunk is screened again, optionally given its own small induced label schema, and run through one model that extracts entities and the relations between them in a single pass; a second model of the same family finds PII. Then the pipeline crosses a line: from here on it sees the whole corpus at once. It canonicalises the drifted type vocabulary, resolves mentions into single entities, assembles an evidence-linked graph — nodes, edges, and hyperedges — stamps centrality and community onto every node, and projects that graph back onto every chunk as a compact context header. Two exports fall out: a knowledge graph as portable files, and a RAG corpus whose text carries a dense embedding, BM25 term statistics, and the denormalised graph columns. At query time a stateless library fetches from your engine, fuses dense and BM25, rescores with the graph columns, appends what graph traversal rescued, and spends a token budget on the result.

The corpus and the graph are one artifact, not two, because the chunks carry the graph's node ids and the graph's edges carry the chunks' evidence. Split them and both halves stop working.

The DAG¶

The five campaign pipelines are the same shape. Two of them (multihop_rag, wiki2multihop) run 14 stages; the three whose schema is induced rather than hand-authored (uda, ohr_bench, vidoseek) run 15, adding induce between content_screen and extract.

The horizontal line between the two subgraphs is load-bearing. Everything above it is document-level: it sees one chunk or one document, never aggregates, and therefore parallelises freely and can be reused unchanged on an incremental delta. Everything below is corpus-level: it consumes the whole corpus to produce its answer. The framework enforces this as an invariant — a chunk-level stage may not depend on a corpus-level one — and the declaration lives in CapabilityDescriptor.level, one field per Capability, checked exhaustively at import.

That invariant is why redaction and extraction both hang off induce rather than off each other, and why enrich (which re-emits the whole chunk stream) is deliberately not treated as an accumulator even though it is corpus-level.

The stage roster¶

Each row is one Capability (a narrow typed interface) fulfilled by one Provider (a plugin resolved by entry-point name at run time). Every Capability's exact input shape, carrier type and alternative Providers are generated from the source into docs/reference/stage-contracts.json and rendered as the stage contract reference — that file, not this page, is the authority on I/O.

Configured values that are simply the code defaults¶

Worth knowing which knobs were actually turned. The chunk budget was: ChunkBudget defaults to 512 / 64 / 16 and the campaign runs 640 / 80 / 8. The hyperedge block was not: weak_hyperedges: true, max_arity: 12, min_tokens: 25, max_tokens: 180, max_window_sentences: 3 is exactly HyperedgeConfig's default. Nor was the enrichment block: max_neighbors_per_entity: 4, max_triples: 6 are the Provider's defaults too. The extraction threshold 0.5 and the embedding dimension 768 are also defaults.

One flag that is not a default, and says so¶

export_corpus sets unsafe_unmasked_corpus: true. The code's default is False, and the YAML records why it was flipped:

Benchmark corpus: PII spans stay recorded in pii_spans/masked_content, but the exported+embedded text is the RAW content. Measured 2026-08-07: with masking on, 82% of 2Wiki/MultiHop-RAG rows carried [PERSON]-style placeholders INCLUDING the article titles the questions target, and the reused M6 musique/hotpot exports are unmasked — so retrieval numbers were neither valid nor cross-comparable.

The detection still runs and every finding is still written; only the substitution into the exported text is skipped, and the Export logs a warning naming the risk once per run. This is what the project's temperament looks like in practice: the unsafe option exists, it is named unsafe_, it is off by default, and the one place it is on carries the measurement that justified it.

The model roster¶

Six models appear in the configured stack. Every licence below is read from the Provider's own ProviderProfile, which records weights and code separately with a citation and a verification date — that is the framework's licence contract, not a summary someone typed.

Two more models are available and were deliberately not in the measured stack — knowledgator/gliner-linker-large-v1.0 behind disambiguation.glinker (Apache-2.0 / Apache-2.0, verified 2026-07-14) and the ULTRA link-prediction weights mgalkin/ultra_3g behind graph_completion.ultra (MIT / MIT, verified 2026-07-09). The campaign ran the deterministic exact_surface resolver and no graph completion at all, so every benchmark number below was produced without inferred edges.

Everything else in the roster is a zero-model Provider: the chunker, both screeners, the profiler, the type consolidator, the exact-surface disambiguator, the graph assembler, the degree feature computer, the context enricher, both exports and the BM25 tokenizer are pure-Python, deterministic and dependency-free, carrying the repository's own Apache-2.0 licence and no weights. That is not an accident of scope — it is the reason a run reproduces byte for byte, and the reason a CPU host can execute most of the DAG.

The licence discipline, stated once. Default reference Providers must have permissive weights and permissive code, verified separately and recorded before they ship (ADR-0012). The one restricted-licence family in the ecosystem — Gemma — is explicitly non-default and opt-in, because its flow-down obligations bind every downstream recipient. The framework's own outbound licence is Apache-2.0 (ADR-0063); NOTICE carries the full provenance.

The deliverable is files¶

Nothing here is a live database. The run's output is a directory (ADR-0017):

• The graph — graph-nodes.parquet and graph-edges.parquet (Arrow/Delta-native), graph.ttl (RDF Turtle for a triple store), graph.graphml (yEd / Gephi / networkx), graph-hyperedges.parquet plus the graph-hyperedges.kv point-lookup sidecar.
• The corpus — corpus.jsonl (streaming, diffable) and corpus.parquet (columnar), every row carrying the chunk text, its provenance, its pii_spans and masked_content, its 768-dim embedding, and the denormalised KG columns (kg_node_ids, neighbor_node_ids, kg_node_community, kg_node_centrality).
• The lexical signal — bm25-stats.json (corpus df / avgdl / Robertson idf) and bm25-postings.parquet (per-record forward postings), written by a tokenizer whose term unit exactly matches the query-side one.
• The Quality Report and the run manifest.

All IO crosses an fsspec seam, so file://, s3://, gcs:// and az:// are one line of configuration apart (ADR-0009). Nothing is locked to a vendor's store, and a downstream consumer needs no framework code to read any of it.

Closing the loop: retrieval¶

The corpus exists to be retrieved from, and the retrieval half holds no index and runs no first-stage search of its own — it composes over candidates your engine returned. The production composition is four steps:

1. BASE — dense + BM25, fused by RRF between those two only, then max-normalised to 1.0.
2. RESCORE — a bounded 1-hop bonus read straight off the KG columns denormalised onto every chunk. No graph database in the path.
3. RESCUE — graph traversal and hypergraph signals contribute candidates the base missed; they are fused with each other, then appended below the rescored base and deduplicated. They never reorder the base. This asymmetry is the whole fix: the earlier design fused graph results in as a co-equal RRF list, and every graph leg lowered recall.
4. SELECT — a token budget is spent over the deduped union by the knapsack packer or the chain-of-evidence selector.

Measured on the full campaign matrix, recall@10, every query, no limit — and read against both baselines, because only one of them is honest:

Graph rescue beats the dense baseline on all seven datasets — but dense is not the strongest rung, and graph_rescue's own first stage already contains BM25. Against bm25, the graph lanes' contribution is within ±0.008 and negative on two. The "+0.068 on uda" is what adding BM25 to dense buys, not what the graph buys.

A dedicated experiment (ticket 06) settled why, and the answer is structural: with a 100–200 candidate base and rescue candidates appended below it, no rescued candidate reaches a top-10 cut at all. An oracle that promotes every rescued gold chunk to the front — which no shippable policy can beat — is worth +0.0056 (multihop_rag), +0.0077 (wiki2multihop) and exactly zero (vidoseek, where the graph lanes reach not one gold chunk the base missed in 1,142 queries). No setting of select_pool_k, k, pool_k or fusion weight converts that into a win.

What these numbers support is the asymmetry itself: applied as rescue, the graph never lowers recall, and fusing it demonstrably does — up to −0.11 at co-equal weight. That containment is real and is why point 3 above is written the way it is. What they do not support is a claim that graph traversal improves retrieval quality here. The chapter on retrieval carries the ceiling tables, the budget curves and the latency story (median 9.8× faster after the optimisation pass; uda graph_rescue 6,648 → 646 ms).

The five corpora, and why one pipeline shape covers them¶

Three of the five arrive as PDFs or page images; two arrive as text. They nevertheless run the same pipeline, for two reasons.

First, the format difference is absorbed by the Parse Capability, not by the pipeline shape. The campaign materialised OCR as its own upstream stage — lightonai/LightOnOCR-2-1B served on vLLM, 142,593 pages rendered, 3,470 documents assembled at 2.2–2.5 pages/second, with 10,240 pages (7%, all in 74 uda documents) recorded as failed rather than quietly dropped — and handed the resulting markdown to parser.plaintext downstream. Swapping parser.plaintext for parser.lighton_vllm in the same YAML would fold that step back in; nothing else in the DAG would move. That is the Capability seam doing its job.

Second, the schema difference is absorbed by one optional stage. Where the domain is known, labels are hand-authored and induce is simply absent (and the run makes zero LLM calls). Where it is not, label_inducer.llm derives each chunk's own small label set. The extractor's interface is identical either way, because induced labels arrive on the chunk record rather than in the extractor's config.

And the split follows the OCR boundary, not anyone's taste. The three datasets that run induction are exactly the three that came through OCR. The campaign generator declares them as a named group — INDUCED_DATASETS = ("ohr_bench", "vidoseek", "uda") against TEXT_DATASETS = ("multihop_rag", "wiki2multihop") — beside a comment that states the reason: the OCR'd group induces under a budget cap, the text-native preps use hand-authored labels. The OCR stage names the identical three (OCR_DATASETS) for the same reason from the other side. So the rule is: a corpus you had to read off a page is a corpus whose schema you do not already know. Both facts are declarations in code, which is why the five YAMLs are generated rather than hand-tuned — and why the header of each one says so.

The scale range is the point worth dwelling on for anyone sizing this: 292 documents to 56,687; 2,434 chunks to 220,129; 809 relations to 90,076. The document-phase cost is a function of worker count, not corpus size; the corpus phase runs once, on already-extracted records rather than on raw text. uda — 220k chunks, 2.6M mentions, 677k hyperedges — is the case that forced the streaming work, and the fixes are recorded rather than smoothed over.

A note on reproducibility, because it changed a published number. Determinism here means PYTHONHASHSEED=0 and pinned BLAS threads. Running the same code, corpus and 1,209 queries at OMP_NUM_THREADS=4 instead of 1 gave 33 queries a different dense pool, 4 a different top-10, and exactly one a different gold-hit count — enough to move a reported recall@10 in the fourth decimal. The contract now refuses to run unless OMP_NUM_THREADS, OPENBLAS_NUM_THREADS and MKL_NUM_THREADS are all pinned to 1. A benchmark that lets the thread scheduler edit its results is not a benchmark, and the fix belongs in the harness rather than in a note asking operators to remember.

Swapping any of it¶

Every row of the stage roster names a Capability, and every Capability has more than one Provider. parser.plaintext can become parser.document, parser.pdfplumber, parser.lighton, parser.lighton_vllm, parser.glm, parser.render or parser.endpoint. disambiguation.exact_surface can become .cascade, .embedding or .glinker. chunk.markdown can become chunk.sentence_window. graph_features.degree can become .pagerank. Your own Provider needs no base class and no import of core — a Capability is a structural typing.Protocol, so anything with the right shape registers under the latence.providers entry-point group and resolves by name (ADR-0004).

The conformance suite is what makes that a guarantee rather than a hope: it drives every Provider through its Capability's case and checks contract adherence, graceful failure, declared licence and declared determinism. Replacing LightOn with docling, or GLiNER2 with your own extractor, cannot break the flow. What it can change is quality — and that stays the responsibility of whoever introduces the Provider. See the conformance guide and the stage contract reference.

Where to go next¶

The chapters. Each takes one segment of the DAG and goes all the way down — the interface, the Provider, the model, every configured value and why it has that value, what flows in and out, what breaks if it is wrong, and which alternatives exist.

• 1 · Ingest and document understanding — source, intake screening, parse, chunk, content screening.
• 2 · Extraction and privacy — schema induction, fused entity/relation extraction, redaction, profiling.
• 3 · From corpus to knowledge graph — type consolidation, disambiguation, graph assembly, hyperedges, graph features, edge prediction.
• 4 · Enrichment and export — the KG header, the twin exports, embeddings and BM25.
• 5 · Retrieval — graph-rescue retrieval, the packer, the measured results.

If you are a developer, read chapters 1 → 4 in order with a pipeline YAML open beside them, then the stage contract reference for exact I/O and the pipeline reference for the config schema. The quickstart gets you a running pipeline first if you would rather start from a green run.

If you are a researcher, the reasoning narrative is the deep dive (why the design is shaped this way, in five acts); this page and its chapters are the exhaustive what and how. The measured evidence lives in the benchmark pages and the campaign guide (running the campaign); the decisions, with their alternatives and their falsifiers, are in the decision log. Chapter 3 and chapter 5 carry the results most likely to be contested.

If you are an architect or evaluating this for an organisation, start with architecture — three orthogonal seams, and what each one buys you — then this page's model roster and licence discipline, then deployment and the threat model. The operational shape in one line: heavy work is document-parallel and independent of corpus size, the corpus-wide pass runs once on extracted records, the output is portable files on any fsspec-addressable store, incremental deltas keep it current without reprocessing, retrieval is stateless and runs on the engine you already own, and every model is swappable behind a conformance-tested interface with its licence declared and dated in code.