Pipeline¶
A Pipeline is a DAG of Stages over a set of input documents, declared once and executable by any Runner (ADR-0003).
pipeline ¶
Declarative Pipeline contract (ADR-0015).
A versioned YAML/JSON document is the canonical Pipeline declaration: an ordered list of Stages, each binding a Capability kind to a Provider (plugin name + config), plus Storage URIs. A typed Pydantic builder constructs and validates the same object, and can round-trip back to YAML — config-as-data that platform teams diff and commit, and what makes the one-line Provider swap literally true.
CapabilityKind ¶
Bases: StrEnum
The Capability a Stage fulfils.
S1 spine: source/parse/export. S3 adds Chunk and Screening's two checkpoints
(intake before Parse, content after Chunk). S4 adds Entity Extraction (typed
entity mentions found in chunked text). S5 adds Relation Extraction (typed
relations between mentions) and a fused Entity+Relation Extraction — one
Provider fulfilling both adjacent Stages in one pass (ADR-0013), so the Pipeline
is not forced to one Provider per Stage. S6 adds Redaction (PII detection +
masked/replaced text variant per document), typically wired off the Chunk Stage.
S7 adds Profiling (corpus-level statistical + quality features), wired off Parse
(per-document features) and, optionally, Entity Extraction (cross-document entity
features). S8 adds Disambiguation (corpus-level entity linking + duplicate merge with
audited, confidence-weighted decisions and Evidence), wired off Entity Extraction
(required — the mentions it resolves) and, optionally, Relation Extraction (the
relations it normalises onto the canonical entities). S9 adds Graph Assembly (corpus-level
knowledge-graph construction — nodes, edges, and per-edge Evidence — from the disambiguated
canonical entities + normalized relations), wired off Disambiguation. W2-linkpred adds the
OPTIONAL Graph Completion Capability (ADR-0037): a post-Assembly link-prediction Stage that
consumes the assembled GraphRecords and appends PREDICTED EDGE-scope GraphRecords, each
UNMISTAKABLY marked properties["inferred"] = True (never merged into the asserted edges,
counted separately in the Quality Report). It depends_on Graph Assembly and is genuinely
optional — a stack without it is byte-identical to today. S9 also adds the
opt-in Embedding Capability (ADR-0017): an Embedder maps RAG-corpus text to a fixed-width
vector so the exported corpus carries embeddings a vector DB can ingest. Embedding is not a
standalone DAG node — it augments the corpus Export: a corpus Export Stage names its Embedder
Provider in config['embedder'] (a {provider, config} block) and the Runner resolves it
and attaches a vector per exported row. It is optional — a corpus Export with no embedder
exports its text unchanged, so any vector DB can still ingest it (the ADR-0017 AC). The
EMBEDDING kind names the seam so a Provider can register + be discovered under it; the
in-core reference is the deterministic embedding.hashing (the learned Granite r2 default
is a pluggable package). W4-schema-induction adds the OPTIONAL SCHEMA_INDUCTION Capability
(ADR-0038): a chunk→chunk Stage inserted after Content Screening and before the extraction
Stages that uses a small OpenAI-compatible LLM to induce a document's own entity/relation/PII
label schema and annotate its chunks (:class:~latence_core.contracts.InducedLabels), which the
extraction Stages UNION onto their config labels. It is genuinely optional — a stack without a
schema_induction Stage is byte-identical to today (the induced-labels field defaults None).
W13-context-enrichment adds the OPTIONAL CONTEXT_ENRICHMENT Capability (ADR-0039): a
corpus-level chunk→chunk Stage placed AFTER Graph Assembly that projects the assembled KG back
onto each chunk as a compact :class:~latence_core.contracts.ContextHeader (its canonical
entities + top-k KG-neighbor triples), which the corpus Export prepends to the EMBEDDING input
ONLY — the stored chunk text stays byte-unchanged. It depends_on the chunk stream AND Graph
Assembly and is genuinely optional — a stack without it is byte-identical to today (the
context_header field defaults None and the Export prepends nothing).
T3 (ADR-0054) adds the OPTIONAL TYPE_CONSOLIDATION Capability: a corpus-level
mention→mention Stage placed between extraction and Disambiguation that canonicalizes the
run's TYPE vocabulary. Per-chunk label induction gives each chunk a small, relevant, uncapped
label set — and produces type drift (org/organization/company), because nothing
forces one schema upstream any more. This Stage folds the drifted labels into canonical
clusters, rewrites every mention's and relation's label (keeping the raw induced label on
raw_label for audit) and emits a :class:~latence_core.contracts.TypeVocabulary. It is
the type-side twin of Disambiguation: consistency is EARNED downstream in the resolution
phase, never IMPOSED upstream by capping a corpus schema. Genuinely optional — a stack
without it is byte-identical to today (raw_label defaults None and no label is rewritten).
ExtractOn ¶
Bases: StrEnum
Which chunk-text variant the extraction Stages read (W18, ADR-0044).
unmasked(default) — extraction reads the raw chunkcontent, so the KG keeps the real entities. Byte-identical to pre-W18: redaction masks only the exported RAG corpus text.masked— extraction reads the REDACTED chunk stream (masked_content), so PII never reaches extraction / the KG at all (the stricter privacy posture some enterprises require). The extraction Stages must depend (transitively) on a redaction Stage somasked_contentis populated before they run — the Pipeline validator enforces this.
Stage ¶
Bases: BaseModel
One node in the Pipeline DAG: a named Stage bound to a Provider.
Pipeline ¶
Bases: BaseModel
A DAG of Stages over a set of input documents, declared once.