Skip to content

Pipeline Stages

A Pipeline is a DAG of Stages over a set of input documents, declared once and executable by any Runner. Each Stage has a typed input and output contract and depends on a Capability, not a named model. Stages fall into two classes: per-document (run independently over each document) and corpus-level (reason across the whole corpus).

flowchart LR subgraph PerDoc["Per-document"] Source --> Parse --> Chunk --> Entity["Entity Extraction"] --> Relation["Relation Extraction"] --> Redaction Intake["Intake Screening"] -.before Parse.-> Parse Content["Content Screening"] -.after Chunk.-> Chunk end subgraph Corpus["Corpus-level"] Profiling --> Disambiguation --> Graph["Graph Assembly"] --> Export end Redaction --> Profiling

Per-document Stages

Source

Lists and fetches documents from where they live — a folder, an object store, or (via connectors) a source system like SharePoint — stamping initial Provenance. The entry point that makes siloed data addressable. v1 ships filesystem/object-store connectors only (ADR-0014). Source produces a ParserInput: the raw, undecoded bytes plus Provenance and Classification (ADR-0019).

Parse

Turns a source document (PDF, image, office file) into markdown plus a page map that preserves original page boundaries, along with an error disposition on the record contract (ADR-0020). OCR is one Provider technique, not the Stage. Providers: parser.plaintext, parser.document, parser.pdfplumber, parser.render (universal, license-clean), and GPU/endpoint OCR (parser.lighton, parser.glm, parser.lighton_vllm, parser.endpoint). See the provider catalog.

Chunk

Splits parsed documents into retrieval-sized pieces while preserving offsets, page alignment, Provenance, and Classification (ADR-0021). Providers: chunk.markdown, chunk.sentence_window, chunk.page (one chunk per source page when the document carries a measured page map, else a 1024-token window — ADR-0064).

Screening

Catches dangerous data early at two checkpoints: Intake Screening before Parse (malware, zip bombs, file-type spoofing, oversized/corrupt files) and Content Screening after Chunk (prompt injection, harmful content, sensitivity escalation). Screening either Quarantines a record — off the DAG, but retained and inspectable — or flags it with risk markers that propagate downstream (ADR-0021). Providers: screening.intake_signature, screening.content_keyword, screening.content_fuzzy.

Entity Extraction

Finds typed entity mentions in text (zero-shot NER and friends). The zero-shot label set lives in config; the deterministic extractor ships in core, learned ones are separate packages (ADR-0022). Entity Extraction emits true original offsets by carrying the chunk's strip offset map forward (ADR-0031). Providers: entity.gazetteer, entity.gliner, entity.endpoint, and the fused fused_entity_relation.gliner2 / fused_entity_relation.gliner25.

Relation Extraction

Finds typed relations between entity mentions within a document. A Fused Provider may emit both entity and relation carriers in one pass, threaded by a mixed-carrier checkpoint (ADR-0023). The Provider is chosen empirically (ADR-0013). Providers: relation.pattern, relation.gliner_relex, relation.llm, and the fused fused_entity_relation.gliner2 / fused_entity_relation.gliner25.

Redaction

Detects PII and produces masked/replaced variants of the text — one RedactionRecord per chunk, keyed off Classification.sensitivity, with an auditable span list (ADR-0024). Redaction is a chunk→chunk transform, closing the document-level truncation PII leak (ADR-0042), over a universal financial-PII floor with an optional mask-before-extract mode (ADR-0044). Providers: redaction.hybrid_rule, redaction.gliner_pii, redaction.gliner2, redaction.presidio, and redaction.gliner25 — whose PII detection is zero-shot on a general GLiNER 2.5 checkpoint and unmeasured, because no GLiNER 2.5 PII checkpoint exists (ADR-0064).

Corpus-level Stages

Profiling

Computes statistical and quality features across the corpus — density, readability, Zipf, cross-document entity frequency, co-occurrence — as scoped FeatureRecords over a fixed feature set, streaming (ADR-0025). Providers: profiling.statistical, profiling.lightweight.

Disambiguation

Links entity mentions to canonical entities and merges duplicates across documents (the two internal steps are entity linking and entity resolution). CPU-viable, over a ported resolver/linking/normaliser cascade with an audited, no-silent-over-merge log (ADR-0026), and an enterprise GLinker neural linker (ADR-0046). Providers: disambiguation.cascade, disambiguation.exact_surface, disambiguation.embedding, disambiguation.glinker.

Graph Assembly

Builds the canonical knowledge graph — nodes, edges, and per-edge Evidence — from disambiguated entities and relations, with deterministic content-addressed ids (ADR-0027). Providers: graph.canonical, graph.weighted.

Export

Materializes the AI-ready outputs — the RAG corpus and the knowledge graph as portable files (Parquet, JSONL, TTL, GraphML), never a live DB (ADR-0017). Providers: export.jsonl_parquet, export.knowledge_graph, export.demo_site.

Optional Stages

These extend the spine and are off by default:

  • Schema Induction — an LLM induces a document's own entity/relation/PII label schema, UNIONed onto the config-label floor (ADR-0038). Provider: label_inducer.llm.
  • Graph Completion — a post-Assembly stage appending predicted edges marked inferred=True, counted separately from asserted edges (ADR-0037). Providers: graph_completion.reference, graph_completion.ultra.
  • Context Enrichment — projects the assembled KG back onto each chunk into the embedding input only (ADR-0039). Provider: context.kg_header.
  • Embedding — opt-in RAG-corpus vectors; see Embedders. Providers: embedding.hashing, embedding.sentence_transformers, embedding.endpoint.

The run's output: the Quality Report

Every run emits a first-class Quality Report — per-Stage metrics, corpus-level KG statistics, contract-completeness checks, and optional gold-set precision/recall; a delta run also reports what changed (ADR-0008, ADR-0028).