Stage contracts — the replacement contract for every Capability¶

This page is generated from the framework's single source of truth — the CapabilityDescriptor table, the Capability Protocols and the record contracts (see Capability protocols and Data contracts for the full API pages) — and a drift gate keeps it byte-identical to those sources. It exists so that a coding agent can replace any component at any Stage — a parser, an extractor, a disambiguator — with full knowledge of what that Stage receives and what it must emit, such that a replacement cannot violate the end-to-end flow.

Machine-readable twin: stage-contracts.json. Regenerate both with PYTHONHASHSEED=0 .venv/bin/latence contracts --write --repo-root ..

Replacing a Provider — the five steps¶

Worked example: replace parser.lighton (the LightOn OCR parser) with your own parser.

Read the contract entry. Find the Capability's section below — for a parser, the parse entry: its Protocol (Parser, one method parse(inputs: Iterable[ParserInput]) -> Iterator[DocumentRecord]), the input shape the Runner delivers, the carrier it must emit, and the wiring rules. The method and carrier docstrings reproduced here ARE the contract — offsets, page maps, Provenance and determinism obligations included.
Implement the Protocol. Capabilities are structural typing.Protocols: any object with the right shape satisfies one — no base class, no import of latence-core required. The end-to-end path is Authoring a Provider; the real template is the example provider.
Declare the entry point in your package's pyproject.toml under [project.entry-points."latence.providers"], named <prefix>.<variant> with the Capability's prefix from its entry below — e.g. parse.myparser = "my_pkg.provider:MyParser". The prefix IS the routing: an unknown prefix is refused loudly at discovery, never silently skipped.
Run the conformance suite. Your Provider is auto-covered by its name prefix; the C1–C6 gate (valid typed records, graceful typed failure, license evidence, determinism-or-documented, device honoured, no PII leak) is the machine check that a replacement holds the contract — see Passing Provider Conformance:

LATENCE_CUDA=0 PYTHONHASHSEED=0 uv run pytest packages/latence-core/tests/test_conformance.py packages/latence-core/tests/test_e2e_conformance.py

Pin it in the pipeline. Name the new Provider in the Stage's provider: field of your stack/pipeline YAML (or let latence setup write it — e.g. its --parser option), and run.

The quality stance: the contract gate guarantees the flow, not the quality — a conformance-green replacement cannot break the pipeline, but the quality of its output is the responsibility of the user who introduces it; measure it with the benchmark harness before trusting it.

How to read an entry¶

The per-Capability facts below are the fields of one frozen CapabilityDescriptor; the field meanings, in the words of the source module:

Everything the framework must know about one Capability, declared once.

Adding a Capability = adding one of these. The exhaustiveness check in index_descriptors makes that mandatory rather than customary.

Fields, and who consumes each:

kind The CapabilityKind this describes. protocol The Capability Protocol a Provider bound to this kind must satisfy — the same check the Runner's dispatch enforces at run time and stacks.capability_guard enforces statically on a CPU host (ADR-0036). carrier The record type a Stage of this kind checkpoints and threads downstream. Record means a MIXED carrier (a fused extractor emits entity and relation mentions; Type Consolidation re-emits both, relabelled) read back polymorphically. None means the kind has no carrier of its own: EXPORT is a pass-through sink whose carrier is its nearest producing ancestor's (see export_carrier_of), and EMBEDDING is not a DAG node at all. level CapabilityLevel — corpus-level vs document-level by kind. Consumed by the PHASE-BOUNDARY invariant and by delta reuse. input_plan StageInputPlan — what a Stage of this kind RECEIVES: the shape the executor delivers its input in, and the wiring rules its input must satisfy (with the exact typed message each raises). Consumed by the Runner's input resolution (StageInputs) and by the offline stack check. accumulates Corpus-level AND its own output is bounded by the knowledge graph rather than by the corpus, so the executor materializes its whole input and holds the result (ADR-0033). This is a strictly narrower fact than level is CORPUS: an EXPORT is a sink (nothing downstream holds its output) and CONTEXT_ENRICHMENT re-emits the entire chunk stream (its output is corpus-sized, so treating it as an accumulator would defeat the streaming release). entry_point_prefixes The latence.providers entry-point name prefixes that name this Capability (entity.gliner → entity). A dotted prefix discriminates on the second segment: screening.intake vs screening.content share a first segment. conformance_case The value of the conformance suite's Capability member this kind is checked under. Several kinds share one case (a fused extractor is driven through the RELATION case). quality_section The QualityReport field this Capability fills, or None when it contributes only per-Stage metrics. GRAPH_COMPLETION shares graph — its predicted edges are counted separately inside GraphQuality and never merged into the asserted-edge statistics (ADR-0037). dag_node False for EMBEDDING, the one Capability that is not a top-level Stage: it is a nested Export sub-config (config['embedder'], ADR-0017). A caller that mis-treats it as a Stage must fail loudly rather than silently skip a check, so dag_protocol_for raises. corpus_when_config The (key, value) in a Stage's config that promotes it to corpus level regardless of kind. Only SCHEMA_INDUCTION declares one (granularity: corpus merges every document's labels into one salience-capped schema and stamps it onto every chunk — an aggregation wherever it sits in the DAG). materializes_for_export Whether an Export wired (transitively) to a Stage of this kind re-emits THIS kind's carrier. False for the kinds that carry records an Export never re-emits: SOURCE and INTAKE_SCREENING thread raw ParserInput s, so an Export above them re-emits what the Parse they must feed produces, and an EXPORT above an EXPORT resolves through to the real producer.

Levels (document vs corpus):

Does a Stage of this Capability see one unit at a time, or the whole corpus?

DOCUMENT — the front half. It sees one chunk or one document and never aggregates, so it can run per-document, be resumed per-document, and its output can be REUSED across a delta for an unchanged document (W17/ADR-0043).

CORPUS — the back half. It consumes the whole corpus (or an artifact built from it) to produce its output, so it cannot run before the front half has finished, its checkpoint spans documents that were never re-extracted, and the PHASE-BOUNDARY invariant forbids a chunk-level Stage from depending on it.

One Capability's level depends on its config rather than its kind — SCHEMA_INDUCTION at granularity: corpus — which is why callers ask CapabilityDescriptor.level_of with the Stage rather than reading CapabilityDescriptor.level directly.

Input shapes (how the Runner delivers a Stage's input):

HOW the Runner delivers a Stage's input — the executor's four-way input decision, declared.

The Runner used to decide this in a four-way if chain over two hand-maintained frozensets, and the dispatch then re-decided it a second time to know whether incoming was safe to materialize. It is a fact about the Capability, so it is declared here with its siblings:

GATHERED The parents' records are concatenated into ONE list (depends_on order). The Capabilities whose Provider materializes the input anyway — the screening outcomes, Relation Extraction's two-carrier partition, Export's file write — so a lazy stream would move no memory and only cost a second pass (ADR-0033's Honest memory boundary). STREAMED The Protocol is Iterable-in and the Provider consumes it exactly ONCE, so the parents' checkpoints are read record-by-record into the Provider: the input transient is O(1) records rather than O(corpus-slice) (audit R2). PER_CARRIER The Stage reads MORE than one carrier out of one depends_on set (Context Enrichment: chunks + graph), and its Provider consumes each exactly once in its own order — so one incoming stream cannot serve it and partitioning one would buffer a side whole-corpus. It receives one INDEPENDENT lazy stream per carrier instead (dogfood-3). ACCUMULATOR A corpus-level accumulator (ADR-0033) whose input IS the whole doc-level output: it streams each carrier from the parents' checkpoints, so nothing corpus-sized is ever a list. Exactly the kinds that declare CapabilityDescriptor.accumulates — a mismatch between the two is refused at import (index_descriptors).

Stage Capabilities¶

`source` — Source¶

Register as: source.<variant> under latence.providers
Level: document — DAG node: true — accumulates: false
Receives (input shape): gathered
Emits (checkpoint carrier): ParserInput
An Export above it re-emits: none
Quality Report section: per-Stage metrics only
Conformance case: source

Protocol `Source`¶

Lists and fetches documents from where they live, stamping Provenance.

produce(self, storage: Storage) -> Iterator[ParserInput]

Yield one raw document per source document, with initial Provenance.

The yielded ParserInput carries the undecoded source content (bytes) — decoding and the turn into markdown belong to the Parser (ADR-0019), not the Source.

Optional refinement — `DocumentEnumerator`¶

OPTIONAL Source refinement: enumerate document ids WITHOUT fetching content.

A delta run has to know which documents the source currently holds before it can decide which of them are unchanged and can therefore be skipped (ADR-0018/0043). Deriving that set from Source.produce is correct but pays the full fetch cost for every document — for a remote Source (a SharePoint library, an object store) that means the whole corpus crosses the network on every delta, which is precisely the cost the delta exists to avoid, and it happens before the exclusions are known, so no exclude_document_ids can prevent it.

A Source that can name its documents more cheaply than it can fetch them implements this: SharePoint reads the content hash Graph reports in the listing, so a delta run over an unchanged library transfers zero bytes. The ids MUST be the same content-addressed ids Source.produce stamps on Provenance.document_id — an id that disagreed would silently mis-classify documents as new. A Source that cannot do better simply does not implement it, and the runner falls back to produce (correct, just not free).

The enumeration MAY be partial, and a Source that can only name some of its documents cheaply MUST leave the rest out rather than fetch them: an omitted id cannot intersect the parent Version, so its document is treated as new and fetched exactly once by produce, whereas fetching it here transfers it unconditionally in addition to that — which is how a no-op delta ends up costing twice a full run (audit R3-connectors_ingestion-8). Omitting can only lose a reuse; it can never invent one, and it never changes a document's records.

document_ids(self, storage: Storage) -> Iterator[str]

Yield the Provenance.document_id of every document currently in the source.

Existing Providers

Provider	Package
`source.local_folder`	`latence-core`
`source.sharepoint`	`latence-source-sharepoint`

`intake_screening` — IntakeScreener¶

Register as: screening.intake.<variant> under latence.providers
Level: document — DAG node: true — accumulates: false
Receives (input shape): gathered
Emits (checkpoint carrier): ParserInput
An Export above it re-emits: none
Quality Report section: screening
Conformance case: intake_screening

Protocol `IntakeScreener`¶

Screens raw documents BEFORE Parse (CONTEXT Screening — intake checkpoint).

Catches malware, zip bombs, file-type spoofing, oversized/corrupt files. A dangerous document is Quarantined (removed from every downstream Stage); a clean one passes through untouched.

screen_intake(self, inputs: Iterable[ParserInput]) -> ScreeningOutcome[ParserInput]

Partition raw inputs into passed vs quarantined, with findings.

Existing Providers

Provider	Package
`screening.intake_signature`	`latence-core`

`parse` — Parser¶

Register as: parse.<variant>, parser.<variant> under latence.providers
Level: document — DAG node: true — accumulates: false
Receives (input shape): streamed
Emits (checkpoint carrier): DocumentRecord
An Export above it re-emits: DocumentRecord
Quality Report section: parse
Conformance case: parse

Protocol `Parser`¶

Turns raw source documents into parsed markdown records.

parse(self, inputs: Iterable[ParserInput]) -> Iterator[DocumentRecord]

Consume raw ParserInputs and yield parsed markdown records.

Existing Providers

Provider	Package
`parser.document`	`latence-parser-document`
`parser.endpoint`	`latence-parser-endpoint`
`parser.glm`	`latence-parser-glm`
`parser.lighton`	`latence-parser-lighton`
`parser.lighton_vllm`	`latence-parser-lighton-vllm`
`parser.pdfplumber`	`latence-parser-pdfplumber`
`parser.plaintext`	`latence-parser-plaintext`
`parser.render`	`latence-parser-render`

`chunk` — Chunker¶

Register as: chunk.<variant> under latence.providers
Level: document — DAG node: true — accumulates: false
Receives (input shape): streamed
Emits (checkpoint carrier): ChunkRecord
An Export above it re-emits: ChunkRecord
Quality Report section: chunk
Conformance case: chunk

Protocol `Chunker`¶

Splits parsed documents into retrieval-sized ChunkRecords (CONTEXT Chunk).

Preserves offsets, page alignment, Provenance and Classification on every emitted chunk. A PARSE_ERROR DocumentRecord (no usable content) yields no chunks.

chunk(self, documents: Iterable[DocumentRecord]) -> Iterator[ChunkRecord]

Consume parsed DocumentRecords and yield ChunkRecords.

Existing Providers

Provider	Package
`chunk.markdown`	`latence-core`
`chunk.page`	`latence-core`
`chunk.sentence_window`	`latence-core`

`content_screening` — ContentScreener¶

Register as: screening.content.<variant> under latence.providers
Level: document — DAG node: true — accumulates: false
Receives (input shape): gathered
Emits (checkpoint carrier): ChunkRecord
An Export above it re-emits: ChunkRecord
Quality Report section: screening
Conformance case: content_screening

Protocol `ContentScreener`¶

Screens chunked content AFTER Chunk (CONTEXT Screening — content checkpoint).

Catches prompt injection, harmful content, sensitivity escalation. A flagged chunk keeps flowing but carries a RiskMarker that survives into the corpus; a chunk deemed dangerous is Quarantined.

screen_content(self, chunks: Iterable[ChunkRecord]) -> ScreeningOutcome[ChunkRecord]

Return chunks (some marked) plus any quarantined chunks and findings.

Existing Providers

Provider	Package
`screening.content_fuzzy`	`latence-core`
`screening.content_keyword`	`latence-core`

`schema_induction` — LabelInducer¶

Register as: label_inducer.<variant> under latence.providers
Level: document — DAG node: true — accumulates: false
Receives (input shape): streamed
Emits (checkpoint carrier): ChunkRecord
An Export above it re-emits: ChunkRecord
Quality Report section: per-Stage metrics only
Conformance case: schema_induction
Config-dependent level: granularity: corpus promotes a Stage of this kind to corpus level

Protocol `LabelInducer`¶

Induces a document's own label schema and annotates its chunks (CONTEXT Classification).

The W4 schema-induction Capability (ADR-0038): an OPTIONAL Stage inserted right after Content Screening and before the extraction Stages. It restores the "point it at a folder and it discovers its own schema" property — a small OpenAI-compatible LLM with structured JSON output reads the document text and induces the entity / relation / PII types actually evidenced in it, then broadcasts that InducedLabels schema onto every ChunkRecord of the document. The extraction Stages then UNION the induced types onto their config labels (the latence_core.induced seam), so a Stage with no config labels becomes fully unsupervised and one with config labels is supervised-plus-augmented — without any extractor rewrite.

It is a chunk→chunk transform (the Content-Screening shape): it consumes the run's chunks and yields the SAME chunks with induced_labels populated (a doc-level Provider groups chunks by document_record_id and makes one LLM call per document by default, broadcasting the doc's induced labels to all its chunks). The Capability names no LLM and no label set — the Provider supplies both. It is fail-open (G1 posture): any inducer failure (network, malformed output, truncation, schema-invalid, empty) leaves a chunk's induced_labels None so the extractor falls back to its config labels and the run never crashes — the untrusted-LLM discipline lives entirely in the Provider. A chunk it could not annotate simply flows through unchanged.

induce(self, chunks: Iterable[ChunkRecord]) -> Iterator[ChunkRecord]

Consume the run's chunks; yield the same chunks with induced_labels set.

The stream is threaded through 1:1 (never dropped or reordered): a chunk the Provider could induce labels for carries a populated InducedLabels; a chunk whose induction failed (fail-open) flows through with induced_labels still None. Deterministic + seeded (temperature 0, stable label order) so a run is byte-reproducible (Baseline bar).

Existing Providers

Provider	Package
`label_inducer.lexicon`	`latence-schema-inducer`
`label_inducer.llm`	`latence-schema-inducer`

`entity_extraction` — EntityExtractor¶

Register as: entity.<variant> under latence.providers
Level: document — DAG node: true — accumulates: false
Receives (input shape): streamed
Emits (checkpoint carrier): EntityMention
An Export above it re-emits: EntityMention
Quality Report section: entities
Conformance case: entity

Protocol `EntityExtractor`¶

Finds typed entity mentions in chunked text (CONTEXT Entity Extraction).

Zero-shot NER: the label set is configured per Pipeline (Stage config), not hardcoded — the Capability names no types. Each emitted EntityMention carries its label, confidence, and the char span it occupies in the parent document's assembled markdown, so a mention resolves back through the parent's page map to its source page(s). An empty or near-empty chunk yields no mentions (never an error).

extract(self, chunks: Iterable[ChunkRecord]) -> Iterator[EntityMention]

Consume ChunkRecords and yield typed EntityMentions.

Existing Providers

Provider	Package
`entity.endpoint`	`latence-ner-endpoint`
`entity.gazetteer`	`latence-core`
`entity.gliner`	`latence-ner-gliner`

`relation_extraction` — RelationExtractor¶

Register as: relation.<variant> under latence.providers
Level: document — DAG node: true — accumulates: false
Receives (input shape): gathered
Emits (checkpoint carrier): RelationMention
An Export above it re-emits: RelationMention
Quality Report section: relations
Conformance case: relation

Wiring rules (violations raise exactly these typed errors):

requires EntityMention (subclass accepted):

Stage {stage} (relation_extraction) received no EntityMentions. A Relation-Extraction Stage must depend on an Entity-Extraction Stage (the mentions it relates). Check the Pipeline's depends_on wiring.
requires ChunkRecord (subclass accepted):

Stage {stage} (relation_extraction) received no ChunkRecords. A Relation-Extraction Stage must also depend on the Chunk (or Content-Screening) Stage for the text + page map. Check depends_on.

Protocol `RelationExtractor`¶

Finds typed relations between entity mentions (CONTEXT Relation Extraction).

Relation Extraction relates mentions within a document (ADR-0022): given the document's EntityMentions plus the chunk text that supplies the surrounding context, it yields directed, typed RelationMentions (head → tail). The relation label set is configured per Pipeline (Stage config), not hardcoded — the Capability names no relation types (zero-shot / prompt-driven), mirroring Entity Extraction. Each emitted relation carries the head/tail mention refs, its label, confidence, and a Provenance span covering both endpoints, so it resolves back through the parent's page map to its source page(s). A document with fewer than two mentions, or no relatable pair, yields no relations (never an error).

relate(self, chunks: Iterable[ChunkRecord], mentions: Iterable[EntityMention]) -> Iterator[RelationMention]

Consume a run's chunks + mentions and yield typed RelationMentions.

Both carriers are supplied: mentions are what a relation connects, and chunks carry the text (and the parent page map) a text-based Provider needs for context and offset→page resolution. The Provider groups both by document_record_id internally — a relation only ever links two mentions of the same document.

Existing Providers

Provider	Package
`relation.gliner_relex`	`latence-relation-gliner`
`relation.llm`	`latence-relation-llm`
`relation.pattern`	`latence-core`

`fused_entity_relation` — FusedEntityRelationExtractor¶

Register as: fused_entity_relation.<variant> under latence.providers
Level: document — DAG node: true — accumulates: false
Receives (input shape): streamed
Emits (checkpoint carrier): Record (mixed: entity + relation mentions in one checkpoint, read polymorphically)
An Export above it re-emits: Record
Quality Report section: relations
Conformance case: relation

Protocol `FusedEntityRelationExtractor`¶

One Provider that fulfils BOTH Entity + Relation Extraction in one pass (ADR-0013).

The Pipeline does not force one Provider per Stage: a fused model (gliner-relex, the custom span-predictor) does joint NER + RE, so a single fused Stage produces both the EntityMentions and the RelationMentions. The Runner routes a fused Stage's FusedExtraction to both downstream consumers without running a separate Entity-Extraction Stage — the S5 seam requirement.

extract_fused(self, chunks: Iterable[ChunkRecord]) -> FusedExtraction

Consume ChunkRecords and return mentions + relations together.

Existing Providers

Provider	Package
`fused_entity_relation.gliner2`	`latence-extract-gliner2`
`fused_entity_relation.gliner25`	`latence-gliner25`
`fused_entity_relation.header_refs`	`latence-core`
`fused_entity_relation.inline_refs`	`latence-core`

`redaction` — PIIDetector¶

Register as: redaction.<variant> under latence.providers
Level: document — DAG node: true — accumulates: false
Receives (input shape): streamed
Emits (checkpoint carrier): ChunkRecord
An Export above it re-emits: ChunkRecord
Quality Report section: redaction
Conformance case: redaction

Protocol `PIIDetector`¶

Detects PII per chunk and produces a masked variant (CONTEXT Redaction; W16, ADR-0042).

The S6 Redaction Capability, made chunk-level in W16 to fix the document-level truncation leak. Before W16 a Provider scanned the WHOLE parsed DocumentRecord, feeding it into a fixed PII model window (768 tokens for the gliner-family) — so any PII past that window was SILENTLY truncated and leaked unmasked into the corpus. Redaction is now a chunk→chunk transform (the Content-Screening / Context-Enrichment shape): a Provider consumes the run's ChunkRecords and yields the SAME chunks with masked_content + pii_spans populated. Each chunk's text (≤ the model window) is PII-scanned + masked in full — nothing is truncated, so PII past the old 768-token document-truncation point is now caught (the ADR-0042 fix).

The chunk's content stays UNMASKED (so a later re-read / audit sees the full entities); the additive masked_content is the PII-safe variant the Export materializes for the RAG corpus, and pii_spans are the chunk-local, counts-only detected spans (offsets into the chunk content, source pages resolved through the chunk's offset_map+page_slice — never the raw value, S6 AC). Redaction is metadata-aware: the Provider keys its policy off each chunk's Classification.sensitivity (S6 AC). The PII type set is Provider/config-supplied — the Capability names no PII types. A chunk with no detected PII yields a chunk whose masked_content equals its content and whose pii_spans is empty (never an error).

The shared policy/masking/no-op-floor/page-resolution machinery lives in latence_core.redaction_policy (plan_chunk_redaction + finalize_redacted_chunk), so a Provider owns ONLY detection. Because Redaction reads the chunk's markup-stripped content, a span's chunk-local offset resolves to its ORIGINAL source page through the chunk's offset_map + the PageIndexResolver — the coordinate-correctness ADR-0031/0042 guarantee. The resolver reads the chunk's OWN page_slice (v19), so resolution is exact in any order and with any sibling chunk missing. A Provider builds ONE resolver per redact call and hands it to every plan_chunk_redaction (it is a required argument, so the page seam cannot be forgotten). Redaction therefore depends on the Chunk Stage (not Parse).

The document-level redact shape (Iterable[DocumentRecord] -> Iterator[RedactionRecord]) is DEPRECATED; a Provider MAY retain it as a secondary redact_documents method for the superseded path, but the blessed Capability is the chunk seam below.

redact(self, chunks: Iterable[ChunkRecord]) -> Iterator[ChunkRecord]

Consume the run's ChunkRecords; yield the SAME chunks with PII masked.

The chunk stream is threaded 1:1 and order-stable (never dropped or reordered): each yielded chunk carries masked_content (its text with every non-TAG PII span replaced) + pii_spans (the chunk-local detected spans) + redaction_disabled_for_sensitive (the H-C1 §4 no-op floor). content is byte-UNCHANGED. Chunks are emitted in first-seen order for determinism; a seeded run is byte-reproducible (Baseline bar).

Existing Providers

Provider	Package
`redaction.gliner2`	`latence-pii-gliner2`
`redaction.gliner25`	`latence-gliner25`
`redaction.gliner_pii`	`latence-pii-gliner`
`redaction.hybrid_rule`	`latence-core`
`redaction.presidio`	`latence-pii-presidio`

`profiling` — Profiler¶

Register as: profiling.<variant> under latence.providers
Level: corpus — DAG node: true — accumulates: true
Receives (input shape): accumulator
Emits (checkpoint carrier): FeatureRecord
An Export above it re-emits: FeatureRecord
Quality Report section: profiling
Conformance case: profiling

Wiring rules (violations raise exactly these typed errors):

requires DocumentRecord (exact carrier):

Stage {stage} (profiling) received no DocumentRecords (got: {got}). A Profiling Stage must depend on the Parse Stage (the documents it profiles). Check the Pipeline's depends_on wiring.

Protocol `Profiler`¶

Computes corpus-level statistical + quality features (CONTEXT Profiling).

The S7 Profiling Capability. Profiling is a corpus-level Stage: it consumes the whole run's parsed DocumentRecords (and, when an Entity-Extraction Stage is upstream, the run's EntityMentions) and emits FeatureRecords — one DOCUMENT-scope record per parsed document carrying its per-document features (density, readability, Zipf α, compression ratio, structure) and exactly one CORPUS-scope record carrying the cross-document aggregate features (entity frequency, co-occurrence, type consensus, source coverage).

Both carriers are supplied so a fused corpus profile can relate document text to the entities extracted from it; a Provider that only wants document text ignores the mentions. The feature pass must be streaming / spill-to-disk friendly — it must not assume the whole corpus fits in RAM (S7 AC) — so a Provider folds documents in one at a time rather than materialising the corpus. A PARSE_ERROR document (no usable content) is skipped, mirroring the Chunk Stage. Records are emitted deterministically (document-scope in first-seen order, then the single corpus-scope record last), so a seeded run is byte-identical.

profile(self, documents: Iterable[DocumentRecord], mentions: Iterable[EntityMention]) -> Iterator[FeatureRecord]

Consume the run's documents + mentions; yield per-document + corpus FeatureRecords.

Existing Providers

Provider	Package
`profiling.lightweight`	`latence-core`
`profiling.statistical`	`latence-core`

`type_consolidation` — TypeConsolidator¶

Register as: type_consolidation.<variant> under latence.providers
Level: corpus — DAG node: true — accumulates: true
Receives (input shape): accumulator
Emits (checkpoint carrier): Record (mixed: entity + relation mentions in one checkpoint, read polymorphically)
An Export above it re-emits: Record
Quality Report section: type_consolidation
Conformance case: type_consolidation

Wiring rules (violations raise exactly these typed errors):

requires EntityMention (exact carrier):

Stage {stage} (type_consolidation) received no EntityMentions (got: {got}). A Type-Consolidation Stage must depend on an Entity-Extraction (or fused) Stage (the mentions whose types it canonicalizes). Check the Pipeline's depends_on wiring.

Protocol `TypeConsolidator`¶

Canonicalizes the corpus's TYPE vocabulary (CONTEXT Resolution; T3).

The counterpart to Disambiguator in the resolution phase: where a Disambiguator resolves entity mentions to canonical entities, a TypeConsolidator resolves the type vocabulary those mentions are labelled with. Both are corpus-level, and both exist because the front half of the pipeline now runs per chunk: each chunk induces its own labels, which is what gives it a small, relevant, uncapped label set — and what makes chunk A say org where chunk B says organization. Consistency is earned HERE, downstream, instead of being imposed upstream by freezing and salience-capping one corpus schema before extraction (which silently dropped rare-but-critical labels).

It is a corpus-level mention→mention transform: it consumes the whole run's EntityMentions (required — the labels it canonicalizes) and, optionally, its RelationMentions, and returns the SAME records with each label rewritten to its canonical form and the raw induced label kept on raw_label for audit — plus the TypeVocabulary it built.

It runs BEFORE Disambiguation, not after: the entity resolver votes a cluster's type from its members' labels and gates its low-precision rungs on type compatibility, so feeding it a drifted vocabulary would both split clusters that belong together and mistype the ones that survive. Canonical types in, consistent entities and consistent graph out.

Determinism is part of the contract: identical inputs must yield an identical vocabulary and identical records (stable clustering, stable canonical election), so a seeded run is byte-identical (Baseline bar).

consolidate(self, mentions: Iterable[EntityMention], relations: Iterable[RelationMention] = ()) -> TypeConsolidation

Consume the run's mentions (+ optional relations); return them remapped + the vocab.

Existing Providers

Provider	Package
`type_consolidation.cascade`	`latence-core`
`type_consolidation.exact_surface`	`latence-core`

`disambiguation` — Disambiguator¶

Register as: disambiguation.<variant> under latence.providers
Level: corpus — DAG node: true — accumulates: true
Receives (input shape): accumulator
Emits (checkpoint carrier): DisambiguationRecord
An Export above it re-emits: DisambiguationRecord
Quality Report section: disambiguation
Conformance case: disambiguation

Wiring rules (violations raise exactly these typed errors):

requires EntityMention (exact carrier):

Stage {stage} (disambiguation) received no EntityMentions (got: {got}). A Disambiguation Stage must depend on an Entity-Extraction (or fused) Stage (the mentions it resolves). Check the Pipeline's depends_on wiring.

Protocol `Disambiguator`¶

Links mentions to canonical entities + merges duplicates (CONTEXT Disambiguation).

The S8 Disambiguation Capability — the algorithmic crown. Disambiguation is a corpus-level Stage: it consumes the whole run's EntityMentions (required — the mentions it resolves and links) and, OPTIONALLY, the run's RelationMentions (to normalise onto the canonical entities), and emits DisambiguationRecords at two scopes:

one ENTITY-scope record per resolved CanonicalEntity — a cross-document merged cluster with its member mentions, an audited, confidence-weighted merge log (no silent over-merge — a below-policy merge is logged not applied), an optional external-KB link (with a graceful fallback for unlinked mentions), and page-accurate Provenance + Evidence back to the source mentions;
one RELATION-scope record per NormalizedRelation — a relation whose endpoints are remapped to canonical entities and whose label is L2 normalised (fuzzy/alias, type-filtered, inverse-detected + direction-swapped).

Both carriers are supplied so relations can be normalised onto the entities resolved in the same pass; a Provider that only disambiguates entities ignores the relations. The mention side is REQUIRED (unlike Profiling's optional mentions) — Disambiguation with no mentions has nothing to resolve. Records are emitted deterministically (all ENTITY records in canonical-text order, then RELATION records), so a seeded run is byte-identical (Baseline bar). The CPU reference path must work with no embedder/KB; a FAISS/GPU blocking Provider is a pluggable option, not required (S8 AC).

chunks is a third, additive, defaulted carrier: the run's ChunkRecords, so a Provider that resolves mentions by their surrounding context (the learned disambiguation.embedding Provider — the pod proved a bare-surface embedding over-merges distinct short surfaces like IBM and Apple, but surface+context separates them cleanly) can locate a mention's parent chunk (by EntityMention.chunk_record_id) and read a bounded window of its content around the mention's offset. It defaults to an empty tuple so the signature is backward-compatible: the algorithmic cascade ignores it (its behaviour is unchanged), a mentions-only run passes no chunks, and any existing Provider that only accepts (mentions, relations) still satisfies the Protocol — the Runner threads chunks only when the parameter exists (W1 rework, #94).

disambiguate(self, mentions: Iterable[EntityMention], relations: Iterable[RelationMention], chunks: Iterable[ChunkRecord] = ()) -> Iterator[DisambiguationRecord]

Consume the run's mentions (+ optional relations + optional chunks); yield records.

Existing Providers

Provider	Package
`disambiguation.cascade`	`latence-core`
`disambiguation.embedding`	`latence-disambig-embedding`
`disambiguation.exact_surface`	`latence-core`
`disambiguation.glinker`	`latence-disambig-glinker`

`graph_assembly` — GraphAssembler¶

Register as: graph.<variant> under latence.providers
Level: corpus — DAG node: true — accumulates: true
Receives (input shape): accumulator
Emits (checkpoint carrier): GraphRecord
An Export above it re-emits: GraphRecord
Quality Report section: graph
Conformance case: graph

Wiring rules (violations raise exactly these typed errors):

input must be NOTHING BUT DisambiguationRecord:

Stage {stage} ({capability}) requires DisambiguationRecord inputs but received: {wrong}. Check the Pipeline's depends_on wiring.

Protocol `GraphAssembler`¶

Builds the canonical knowledge graph from disambiguated records (CONTEXT Graph Assembly).

The S9 Graph Assembly Capability. Graph Assembly is a corpus-level Stage: it consumes the whole run's DisambiguationRecords (the canonical entities and normalized relations the S8 Disambiguation Stage produced) and emits GraphRecords at two scopes:

one NODE-scope record per CanonicalEntity — a graph node with a deterministic content-addressed id, its member mentions, source-document reach, an optional external-KB link, and Evidence back to the source mentions;
one EDGE-scope record per NormalizedRelation — a graph edge with a deterministic id, its head/tail node ids, the normalized label, and per-edge Evidence back to the source relation's mentions/documents (every edge carries Evidence, the S9 acceptance criterion).

An edge is emitted only when both its endpoints resolved to a node (endpoint validation — no dangling edges). Records are emitted deterministically (all NODE records in node-id order, then EDGE records in edge-id order), so a seeded run is byte-identical (Baseline bar) and two runs over the same corpus produce the same graph. The reference Provider is pure-Python, no embedder/KB required; a graph-DB-writer or embedding-augmented Provider is a pluggable option behind this same seam (ADR-0017: Export writes files, never a live DB).

assemble(self, records: Iterable[DisambiguationRecord]) -> Iterator[GraphRecord]

Consume the run's DisambiguationRecords; yield GraphRecords (nodes then edges).

Existing Providers

Provider	Package
`graph.canonical`	`latence-core`
`graph.weighted`	`latence-core`

`graph_completion` — GraphCompleter¶

Register as: graph_completion.<variant> under latence.providers
Level: corpus — DAG node: true — accumulates: true
Receives (input shape): accumulator
Emits (checkpoint carrier): GraphRecord
An Export above it re-emits: GraphRecord
Quality Report section: graph
Conformance case: graph_completion

Wiring rules (violations raise exactly these typed errors):

input must be NOTHING BUT GraphRecord:

Stage {stage} (graph_completion) requires GraphRecord inputs but received: {wrong}. A Graph Completion Stage must depend on the Graph Assembly Stage (the assembled nodes + edges it completes). Check the Pipeline's depends_on wiring.

Protocol `GraphCompleter`¶

Predicts missing KG edges over the assembled graph (CONTEXT Graph Assembly; W2-linkpred).

The post-Assembly link-prediction Capability (ADR-0037): an OPTIONAL corpus-level Stage that consumes the whole run's assembled GraphRecords (NODE + EDGE) and yields ADDITIONAL EDGE-scope GraphRecords for predicted edges — the §19 predicted-vs-extracted split GraphAssembler carried as is_predicted (ADR-0027) turned into its own Stage. It never re-emits the input nodes/edges and never mutates an asserted edge; it only appends new predicted edges.

A predicted edge is a GraphEdge at scope=EDGE, UNMISTAKABLY marked properties["inferred"] = True (plus scorer/score/rank/ calibrated) so it is never silently merged into the asserted stream — the enterprise-honesty crux (ADR-0037). Its Evidence is honest and model-derived, NOT fabricated mention offsets: mention_ids = [] (there is no textual mention to cite), document_ids = the union of the head + tail nodes' source_document_ids, snippet a human justification naming the scorer + score, and confidence the calibrated score. A predicted edge that COLLIDES (same endpoints + label) with an asserted edge is DROPPED — it is already asserted — logged, not emitted.

Mirroring GraphAssembler: corpus-level, Iterable in / Iterator out, deterministic + seeded → byte-identical (predicted edges yielded in edge-id order), the CPU reference (graph_completion.reference) pure-Python, no torch. A learned ULTRA/PyKEEN scorer is a pluggable Provider behind this same seam (Slice 2). The Stage is genuinely OPTIONAL: a stack with no graph_completion Stage behaves exactly as today.

complete(self, records: Iterable[GraphRecord]) -> Iterator[GraphRecord]

Consume the assembled graph (NODE + EDGE); yield ADDITIONAL predicted EDGE records.

Never re-emits the input, never mutates asserted edges — only appends predicted EDGE-scope GraphRecords, each marked properties["inferred"] = True with honest model-derived Evidence, in edge-id order (deterministic). A predicted edge colliding with an asserted edge (same endpoints + label) is dropped, not emitted.

Existing Providers

Provider	Package
`graph_completion.reference`	`latence-core`
`graph_completion.ultra`	`latence-linkpred-ultra`

`context_enrichment` — ContextEnricher¶

Register as: context.<variant> under latence.providers
Level: corpus — DAG node: true — accumulates: false
Receives (input shape): per_carrier
Emits (checkpoint carrier): ChunkRecord
An Export above it re-emits: ChunkRecord
Quality Report section: context_enrichment
Conformance case: context_enrichment

Wiring rules (violations raise exactly these typed errors):

requires ChunkRecord (subclass accepted):

Stage {stage} (context_enrichment) received no ChunkRecords (got: {got}). A Context-Enrichment Stage must depend on the Chunk (or Content-Screening / Schema-Induction) Stage — the chunks it enriches. Check the Pipeline's depends_on wiring.
requires GraphRecord (subclass accepted):

Stage {stage} (context_enrichment) received no GraphRecords (got: {got}). A Context-Enrichment Stage must also depend on the Graph Assembly Stage — the assembled KG it projects onto each chunk. Check the Pipeline's depends_on wiring.

Protocol `ContextEnricher`¶

Projects the assembled KG back onto each chunk as a compact header (W13 context enrichment).

The W13 context-enrichment Capability (ADR-0039): an OPTIONAL Stage placed AFTER Graph Assembly that stamps each ChunkRecord with a compact ContextHeader — its canonical entities + top-k KG-neighbor triples — which the Export prepends to the EMBEDDING input ONLY. The connections a chunk needs for retrieval relevance are ALREADY computed as the KG; this surfaces them into the vector + a metadata column, without inflating the stored chunk text (the LLM window stays the clean chunk).

Shaped like LabelInducer (a chunk→chunk transform that yields the SAME chunks with an additive field populated via chunk.model_copy(update=...)) but corpus-level and multi-input, like Disambiguator: it consumes the run's ChunkRecords (what it enriches + passes through), EntityMentions (a chunk's canonical entities, via EntityMention.chunk_record_id) AND the assembled GraphRecords (NODE + EDGE — the entities' KG neighbors). It yields the SAME chunks, 1:1 and order-stable, with context_header populated (or left None for a chunk with no gated canonical entity — nothing to add). Genuinely optional + contract-preserving: offsets/provenance/content are untouched, so a stack without the Stage is byte-identical (the field stays None and the Export prepends nothing).

Deterministic + seeded (the CPU reference context.kg_header is pure-Python, no model, no key) → byte-identical: every collection in the header is sorted, and the neighbor triples are confidence-sorted then CAPPED (max_neighbors_per_entity top-k per entity + an overall max_triples) so a corpus-wide hub entity contributes only its TOP relations — the mandatory anti-bloat + hub-entity guard (ADR-0039). The optional LLM situating-sentence mode is a documented follow-on (deterministic=False), not the default.

enrich(self, chunks: Iterable[ChunkRecord], mentions: Iterable[EntityMention], graph_records: Iterable[GraphRecord]) -> Iterator[ChunkRecord]

Consume the run's chunks + mentions + assembled graph; yield the SAME chunks enriched.

The chunk stream is threaded 1:1 and order-stable (never dropped or reordered): a chunk with at least one confidence-gated canonical entity carries a populated ContextHeader; a chunk with none flows through with context_header still None (byte-identical to no-stage for that chunk). Deterministic + seeded → byte-reproducible (sorted + capped + content-addressed, the Baseline bar).

Existing Providers

Provider	Package
`context.kg_header`	`latence-core`

`embedding` — Embedder¶

Register as: embedding.<variant> under latence.providers
Level: document — DAG node: false — accumulates: false
Receives (input shape): gathered
Emits (checkpoint carrier): none
An Export above it re-emits: none
Quality Report section: per-Stage metrics only
Conformance case: embedding

Protocol `Embedder`¶

Optionally embeds RAG-corpus text into vectors (CONTEXT — an opt-in Export augmentation).

The S9 opt-in Embedder Capability (ADR-0017: "embeddings are optional (an opt-in Embedder Provider) so any vector DB can ingest it"). It is not required for Export — the RAG-ready corpus exports its cleaned/chunked/PII-handled text with full Provenance/Classification whether or not an embedder is wired. When a Pipeline opts in, an Embedder maps each text to a fixed-width vector so the exported corpus carries embeddings a vector DB can ingest directly.

The reference embedder in core is a deterministic, dependency-free hashing embedder (no model, no weights, no license to verify — ADR-0012/0016), so the seam is exercised end to end offline; the default learned Embedder (IBM Granite Embedding r2, Apache-2.0; ADR-0045) is a pluggable Provider package, not a core dependency. The Capability names no model — only the seam.

dimension: int

The fixed embedding width every returned vector has.

embed(self, texts: Iterable[str]) -> Iterator[list[float]]

Map each input text to a dimension-wide vector, in input order.

Existing Providers

Provider	Package
`embedding.endpoint`	`latence-embedder-endpoint`
`embedding.hashing`	`latence-core`
`embedding.sentence_transformers`	`latence-embedder-st`

`export` — Export¶

Register as: export.<variant> under latence.providers
Level: corpus — DAG node: true — accumulates: false
Receives (input shape): gathered
Emits (checkpoint carrier): none
An Export above it re-emits: none
Quality Report section: per-Stage metrics only
Conformance case: export

Protocol `Export`¶

Materializes AI-ready outputs (JSONL + Parquet in S1).

A corpus Export Provider may optionally attach embeddings (ADR-0017: an opt-in Embedder augmentation). It opts in by naming an Embedder Provider in its own Stage config['embedder'] (a {"provider": ..., "config": {...}} block); the Provider resolves that Embedder and threads each exported record's text through it, adding a vector per row. A Provider that names no embedder exports its text unchanged — embeddings are never required for Export. The seam is one signature so a non-embedding Export (the KG / demo-site Providers) satisfies it identically.

export(self, records: Iterable[Record], storage: Storage, out_dir: str) -> list[str]

Write records to out_dir on storage; return the output URIs.

Existing Providers

Provider	Package
`export.demo_site`	`latence-demo`
`export.jsonl_parquet`	`latence-core`
`export.knowledge_graph`	`latence-core`

Signal seams (non-Stage Capabilities)¶

These Capabilities register under the same latence.providers group and pass the same conformance gate, but they are not DAG Stages: no CapabilityKind, no checkpoint of their own. They are resolved from a Stage's config (an Export's embedders and tokenizer, Graph Assembly's graph_features) or by the engine (the delta processor).

`sparse` — SparseEmbedder¶

Register as: sparse.<variant> under latence.providers
Conformance case: sparse

Protocol `SparseEmbedder`¶

Encodes corpus text into a sparse term-weight vector (a Signal generator; ADR-0053).

The index-time SPLADE-family sibling of Embedder for the shape a dense vector cannot express: a variable-support term-weight map over a fixed vocabulary, emitted at Export as the parallel sparse_indices / sparse_values columns a store's sparse index ingests directly (ADR-0050). Like every signal generator it is a pure transform that holds nothing (ADR-0048) and names no model — the framework names the seam.

The blessed real Provider (ADR-0053) composes a multilingual SPLADE model's term-weights with the pipeline's confidence-gated NER entity terms + selected metadata, over the redacted corpus text by default; that composition + its license-verified model ship as a pluggable package (latence-splade), NOT in core. The in-core sparse.hashing reference is a deterministic, dependency-free hashing sparse encoder that exercises the seam + emission path offline (no model, no weights, no license to verify — ADR-0012/0016), exactly as embedding.hashing does for the dense seam. It is a reference, not SPLADE.

encode_sparse(self, texts: Iterable[str]) -> Iterator[SparseVector]

Map each text to a SparseVector (canonical ascending indices), in input order.

One vector per input text, in order (like Embedder.embed). Empty/degenerate text yields the empty SparseVector() (never a crash). A deterministic Provider is byte-reproducible (the Baseline bar).

vocab_size: int

The size of the term-index space: every emitted index lies in [0, vocab_size).

Optional refinement — `ComposingSparseEmbedder`¶

A SparseEmbedder that also composes structured per-record signals (ADR-0053).

The index-time seam for the ADR-0053 structured composition. A plain SparseEmbedder exposes only the text-only encode_sparse (the model leg); a composing embedder additionally accepts, per record, the SparseSignals (confidence-bearing entity surfaces + selected metadata) and folds them into the emitted vector — so the corpus's sparse space carries the SAME boosted entity/metadata terms the query-side encoder injects (the one-composition-two-clocks guarantee). Export detects this Capability and passes each row's signals; an embedder that does NOT implement it (e.g. the deterministic sparse.hashing reference) is driven text-only, byte-identically to today.

An implementer is also a SparseEmbedder (it keeps the text-only encode_sparse for the no-signal path); this Protocol declares only the additional composing method so an isinstance check cleanly discriminates the two.

encode_sparse_composed(self, texts: Iterable[str], signals: Sequence[SparseSignals]) -> Iterator[SparseVector]

Map each (text, signals) pair to its composed SparseVector, in input order.

signals is parallel to texts (one SparseSignals per text, same length + order). One vector per input, in order; a record whose signals are EMPTY_SPARSE_SIGNALS composes to exactly the model leg (identical to SparseEmbedder.encode_sparse on that text — the additive guarantee).

Existing Providers

Provider	Package
`sparse.hashing`	`latence-core`
`sparse.splade`	`latence-splade`

`multivector` — MultiVectorEmbedder¶

Register as: multivector.<variant> under latence.providers
Conformance case: multivector

Protocol `MultiVectorEmbedder`¶

Encodes text into a variable-length multi-vector — one vector per token (ADR-0050).

The index-time generator for the ColBERT-style late-interaction shape: a text becomes a sequence of fixed-width token vectors (not one pooled vector). Raw multi-vectors are heavy and have a different access pattern, so Export writes them to a sidecar multivectors.parquet (list<list<double>> keyed by record_id), never into records.parquet (ADR-0050). They also feed an FdeConverter to produce the dense fde_embedding column.

Experimental and off by default (ADR-0050). The in-core multivector.hashing reference is a deterministic, dependency-free per-token hashing encoder that exercises the seam offline; a real ColBERT-family encoder is a pluggable package.

dimension: int

The fixed width of every token vector in every returned multi-vector.

encode_multivector(self, texts: Iterable[str]) -> Iterator[list[list[float]]]

Map each text to a list of dimension-wide token vectors, in input order.

One multi-vector per input text, in order. Empty/degenerate text yields an empty list (no tokens, never a crash). Every token vector has width dimension. A deterministic Provider is byte-reproducible.

Existing Providers

Provider	Package
`multivector.hashing`	`latence-core`

`fde` — FdeConverter¶

Register as: fde.<variant> under latence.providers
Conformance case: fde

Protocol `FdeConverter`¶

Converts a multi-vector into one fixed-width dense vector (MUVERA FDE; ADR-0050).

A data-oblivious, training-free reduction that maps a variable-length multi-vector to a single dimension-wide dense vector, so late-interaction quality can be approximated inside a customer's ordinary dense index with no special store — emitted at Export as the fde_embedding column (ADR-0050). Pure transform, holds nothing, names no model.

Experimental and off by default (ADR-0050): the production converter decision (canonical MUVERA vs a regularizer-refined variant) is deferred, and the quality-blessed canonical MUVERA reference is a pluggable Provider (fde.muvera). The in-core fde.reference is a deterministic, dependency-free SimHash space-partition-and-sum converter that exercises the seam + emission path offline — a genuine data-oblivious FDE (NOT a passthrough that lies, the anti-false-green bar), without claiming a MaxSim-approximation quality guarantee on any corpus.

convert(self, multivectors: Iterable[list[list[float]]]) -> Iterator[list[float]]

Map each multi-vector to one dimension-wide dense vector, in input order.

One dense vector per input multi-vector, in order. An empty multi-vector yields the zero vector of width dimension. A token vector whose width does not match the converter's declared input width is rejected with a typed error (the faithful contract a real converter enforces — anti-false-green). A deterministic Provider is byte-reproducible.

dimension: int

The fixed width of the produced dense vector, independent of the multi-vector length.

Existing Providers

Provider	Package
`fde.muvera`	`latence-muvera`
`fde.reference`	`latence-core`

`tokenizer` — Tokenizer¶

Register as: tokenizer.<variant> under latence.providers
Conformance case: tokenizer

Protocol `Tokenizer`¶

Splits text into lexical terms — the varying seam of the BM25 term-stats export (ADR-0048).

BM25 is a lexical signal, so its corpus statistics (document frequency, term frequency, document length, average document length) are a pure function of one thing that genuinely varies: how text is split into terms. This Capability is that seam. Everything else in the Bm25Accumulator — the df/tf fold, the Robertson idf, the ascending canonical order — is fixed math; the tokenisation is the knob a corpus tunes. Like every retrieval-tooling piece it is a pure transform that holds nothing (ADR-0048) and names no model — the framework names the seam.

Two in-core reference adapters ship behind it (latence_core.bm25), and they are genuinely different tokenisations (not a clone + a rename — the anti-false-green bar for a real seam): tokenizer.regex folds to lowercase Unicode alphanumeric runs (punctuation and the underscore are boundaries and are dropped, matching the dense/sparse hashing references — so Müller and Таможенный are whole terms, though an unspaced script like Chinese yields one term per whitespace run: the reference is Unicode-aware, not a word segmenter, and a corpus that needs segmentation plugs it in HERE), while tokenizer.whitespace splits only on whitespace and keeps punctuation attached to the token (so "foo," and "foo" are distinct terms). A corpus that wants a store's exact analyzer plugs its own Tokenizer in behind this same seam. Both references are deterministic, so the emitted BM25 artifact is byte-reproducible (the Baseline bar).

tokenize(self, texts: Iterable[str]) -> Iterator[list[str]]

Map each text to its list of terms, in input order (one token list per input text).

One token list per input text, in order (like Embedder.embed). The terms appear in their order of occurrence in the text (a BM25 term frequency counts occurrences, so order is immaterial to the statistic but the contract is occurrence-order, not sorted). Empty or term-free text yields the empty list [] (never a crash). A deterministic Tokenizer is byte-reproducible (the Baseline bar).

Existing Providers

Provider	Package
`tokenizer.regex`	`latence-core`
`tokenizer.whitespace`	`latence-core`

`graph_features` — GraphFeatureComputer¶

Register as: graph_features.<variant> under latence.providers
Conformance case: graph_features

Protocol `GraphFeatureComputer`¶

Computes centrality + community per node over the assembled KG (CONTEXT Graph Assembly).

The index-time graph-feature enrichment Capability (ADR-0051): a pure transform that, given the assembled graph's GraphNodes and GraphEdges, returns one GraphFeatures — a centrality measure (degree / PageRank) and a community id per node. It holds no index, runs no search (ADR-0048), names no model. The Graph-Assembly Stage OPTIONALLY resolves a computer from its config['graph_features'] (the same {"provider", "config"} composition Export uses for its Embedder) and stamps the returned features onto each node's properties bag, so they flow into graph-nodes.parquet columns and — via Context Enrichment — onto each chunk's ContextHeader (the two consumers ADR-0051 names: graph-augmented retrieval and the Knapsack packer's centrality / cluster_ids inputs). Genuinely optional: a Stage that names no computer emits the graph byte-identically to today.

Behaviour VARIES behind the seam, so it ships two in-core reference adapters (latence_core.stages.graph_features): graph_features.degree (degree centrality + connected-components community) and graph_features.pagerank (PageRank centrality + label-propagation community). Both are pure-Python, zero-dependency and DETERMINISTIC (fixed iteration order, rounded scores, content-addressed community ids), so a seeded run is byte-reproducible. They are references, not the blessed Louvain/Leiden community detection (ADR-0051), which — like SPLADE/MUVERA — ships as a pluggable, license-verified package.

compute_features(self, nodes: Iterable[GraphNode], edges: Iterable[GraphEdge]) -> GraphFeatures

Compute one GraphFeatures (centrality + community per node) over the KG.

Consumes the whole assembled graph (all nodes + all edges); returns a GraphFeatures covering every input node (never a partial map). An edge whose endpoint is not among nodes is ignored (belt-and-braces — the assembler validates endpoints). An empty graph yields the empty GraphFeatures. Deterministic → a seeded run is byte-reproducible (the Baseline bar).

Existing Providers

Provider	Package
`graph_features.degree`	`latence-core`
`graph_features.pagerank`	`latence-core`

`delta` — DeltaProcessor¶

Register as: delta.<variant> under latence.providers
Conformance case: delta

Protocol `DeltaProcessor`¶

Applies a Delta to the parent corpus with affected-set recompute (CONTEXT Delta).

The S11 Delta Capability (ADR-0018). A DeltaProcessor is model-agnostic and pure over records: given the parent Corpus Version's corpus-level derived records (DisambiguationRecords + GraphRecords), the current run's freshly-extracted corpus-level records, and the classified Delta, it returns a DeltaOutcome: the new live derived set for Version N+1 (parent records the delta did not touch, spliced with the affected-set recompute), the retracted set, and the entity/edge churn counts.

The DeltaProcessor owns the corpus-level recompute of the committed record set — the blocking-neighborhood affected set for Disambiguation and the graph patch by source_document_ids (ports graph/patcher). (Doc-level Stages, Parse … Redaction, run only on the NEW/CHANGED documents since W17/ADR-0043 — an unchanged document's records are reused across runs and the processor consumes the union of reused + freshly-extracted corpus-level records exactly as before, so the seam here is unchanged.) When the Delta signals a full Reconciliation (drift crossed the threshold), the processor re-resolves ALL blocks rather than only the affected set; the seam is identical, only the affected set differs. The reference Provider is pure-Python, deterministic and CPU-viable (the blocking-neighborhood compute_affected_set closure + GraphPatcher); a distributed blocking Provider is a pluggable upgrade behind this same seam.

apply_delta(self, parent_records: Iterable[Record], current_records: Iterable[Record], delta: Delta) -> DeltaOutcome

Apply delta to the parent corpus; return the patched live set + churn.

Existing Providers

Provider	Package
`delta.affected_set`	`latence-core`

Protocol result types¶

The frozen dataclasses several Protocols return — the non-Record half of the output API. Each docstring names its producer.

`ScreeningOutcome`¶

The result of a Screening checkpoint: what passed, what was Quarantined.

passed are the records that proceed downstream — for Content Screening these may carry newly-attached risk markers. quarantined are the QuarantineRecords removed from the pipeline (retained for audit, never exported to the corpus). findings is the full per-record decision log Screening emits into the Quality Report — every Quarantine reason and every flag, so the disposition is auditable (S3 AC).

Field	Type
`passed`	`list[_Screened]`
`quarantined`	`list[QuarantineRecord]`
`findings`	`list[ScreeningFinding]`

`FusedExtraction`¶

The joint output of a FusedEntityRelationExtractor — one pass, both Stages.

A Fused Provider (ADR-0013: gliner-relex, the custom span-predictor) fulfils Entity Extraction AND Relation Extraction in a single pass over the chunks, so it returns both carriers together: the mentions it found and the relations between them. The Runner threads both downstream from the one fused Stage — S4's mentions and S5's relations — without invoking a separate Entity-Extraction Stage (no double-extraction).

Field	Type
`mentions`	`list[EntityMention]`
`relations`	`list[RelationMention]`

`TypeConsolidation`¶

The output of a TypeConsolidator — remapped records + the vocabulary that did it.

Three things travel together because they are one decision: the mentions and relations with their label rewritten to canonical form (and their raw induced label preserved on raw_label), and the TypeVocabulary that defines the mapping. The Runner threads the records downstream and persists the vocabulary beside the Stage's checkpoint — so the consolidation is inspectable as data, and an incremental run can merge into it (ADR-0017: files are the source of truth).

Field	Type
`mentions`	`list[EntityMention]`
`relations`	`list[RelationMention]`
`vocabulary`	`TypeVocabulary \\| None`

`DeltaOutcome`¶

The result of applying a Delta to the parent corpus (S11 affected-set recompute).

A DeltaProcessor takes the last committed Corpus Version's corpus-level derived records, the current run's freshly-extracted corpus-level records, and the classified Delta, and returns:

records — the new live derived set for Version N+1: the parent records the delta did not touch, spliced with the affected-set recompute's freshly-resolved records (the merge-on-add / split-on-delete result). This is what the Corpus Version commits.
retracted — the derived records tombstoned by this delta (soft-retained on a Retraction, discarded on a Purge), so the audit knows exactly what left the live set.
entities_created / entities_merged / entities_split — the affected-set recompute's effect on canonical entities vs the parent (a bridging doc that MERGED two clusters, a deleted doc that SPLIT one) — the S11 churn the Quality Report reports.
edges_added / edges_retracted — the graph patch counts.
affected_set_size / corpus_record_count — the perf-baseline witness: how many corpus-level records the affected-set recompute actually re-resolved (affected_set_size) out of the whole current corpus-level set (corpus_record_count). On an incremental (non-reconciliation) delta affected_set_size is the blocking-neighborhood closure the recompute touched, which is <= corpus_record_count — the measurable "recompute only the affected set" bound (ADR-0018). On a full Reconciliation the two are equal (the whole corpus is the affected region).

Field	Type
`records`	`list[Record]`
`retracted`	`list[Record]`
`entities_created`	`int`
`entities_merged`	`int`
`entities_split`	`int`
`edges_added`	`int`
`edges_retracted`	`int`
`affected_set_size`	`int`
`corpus_record_count`	`int`

`GraphFeatures`¶

Per-node structural features over the assembled KG: centrality + community (ADR-0051).

The result a GraphFeatureComputer returns: for every node in the assembled graph, a centrality score and a community id. It is a value the Graph-Assembly Stage stamps onto each GraphNode's properties bag (centrality / community) — the computer holds nothing and mutates nothing (ADR-0048). Invariants (enforced at construction so a Provider cannot emit a half-populated result, so the downstream stamp is total):

centrality and community cover the same node-id set — every node gets both a centrality and a community, never one without the other (the anti-false-green contract: a computer that skips nodes fails here, not silently downstream);
each centrality value is a finite float (the normalisation is the Provider's business — degree-centrality lands in [0, 1], PageRank sums to ~1 — but every value is a real number, never NaN/inf);
each community id is a non-empty string — the deterministic, content-addressed id of the node's community (the reference Providers use the community's lexicographically-smallest node_id), so two runs over the same corpus agree byte-for-byte (the Baseline bar).

An empty graph yields the empty GraphFeatures (both maps empty), never an error.

Field	Type
`centrality`	`dict[str, float]`
`community`	`dict[str, str]`

`SparseVector`¶

A sparse term-weight vector — the index-time SPLADE-family signal (ADR-0050/0053).

The shape every vector DB's sparse index expects: two parallel columns, indices (the term ids that have non-zero weight) and values (their weights), materialised at Export as sparse_indices: list<int> + sparse_values: list<float> (ADR-0050). Invariants (enforced at construction so a Provider cannot emit a malformed vector, and so the emission is byte-deterministic regardless of a Provider's internal dict ordering):

indices and values are the same length;
indices are strictly ascending (hence unique) and non-negative — the canonical order a sparse index ingests, and the order two runs must agree on for a byte-identical export;
a zero vector (empty text, no surviving terms) is the empty SparseVector() — never a dense run of zeros.

A SparseEmbedder Provider builds one via from_terms from its natural {term_id: weight} map, which sorts and validates in one place.

Field	Type
`indices`	`list[int]`
`values`	`list[float]`

`SparseSignals`¶

The per-record structured signals a composing SparseEmbedder injects at index time.

ADR-0053's sparse vector is not plain SPLADE(text) but the structured composition SPLADE(text) ⊕ boost·entity_terms ⊕ boost·metadata_terms. The text-only SparseEmbedder.encode_sparse seam can only build the model leg — no structured signal crosses it — so a ComposingSparseEmbedder receives this parallel per-record carrier and folds the chunk's own structured signals into the emitted vector. That is what makes the index-time and query-time sparse spaces match (the ADR-0053 one-composition-two-clocks guarantee): without it the query encoder injects boosted entity/metadata terms the corpus never carried, and a hybrid search on an injected term matches zero documents. A record with no structured signals is EMPTY_SPARSE_SIGNALS and composes to exactly the model leg — identical to the text-only path (so the additive/byte-identical guarantee is untouched when no signals exist).

entities — the chunk's (surface, confidence) entity mentions offered to the composition. The composition confidence-gates them; the embedder's redaction gate additionally drops any surface that no longer occurs in the (redacted) corpus text, so a masked PII surface is never re-injected as a term (ADR-0053, REDACTION-BY-DEFAULT).
metadata — the chunk's {field: value} map from which the embedder's operator-selected metadata_fields are injected (e.g. its classification category).

Field	Type
`entities`	`tuple[tuple[str, float], ...]`
`metadata`	`Mapping[str, str]`

Record carriers¶

The Pydantic contracts that flow between Stages. Every carrier extends Record (Provenance + Classification are required — a record cannot lose its lineage at a Stage boundary). Nested value types (page maps, offset maps, Evidence, spans) are documented on the full Data contracts page.

`Record`¶

Base inter-Stage record: always carries Provenance and Classification.

Both are required with no default, so constructing a record without them raises pydantic.ValidationError — the enforced Stage-boundary check.

Field	Type	Required
`schema_version`	`int`	no
`record_id`	`str`	yes
`provenance`	`Provenance`	yes
`classification`	`Classification`	yes

`Provenance`¶

The immutable chain from a record back to its source.

Required fields identify the source document unambiguously; the optional offset fields locate a derived record within it (populated by Parse/Chunk).

Field	Type	Required	Description
`schema_version`	`int`	no
`source_uri`	`str`	yes	Storage URI read from.
`file_name`	`str`	yes
`file_type`	`str`	yes	Lowercased extension, no dot.
`file_size`	`int`	yes	Source size in bytes.
`document_id`	`str`	yes	Content-addressed source doc id.
`source_system`	`str \\| None`	no	Originating system.
`page_start`	`int \\| None`	no
`page_end`	`int \\| None`	no
`char_start`	`int \\| None`	no
`char_end`	`int \\| None`	no

`Classification`¶

Descriptive attributes of a document's content.

Attached at ingest / after Parse and inherited by every downstream record.

Field	Type	Required	Description
`schema_version`	`int`	no
`language`	`str`	yes	BCP-47-ish tag; 'und' if unknown.
`category`	`str \\| None`	no	Content category, e.g. 'contract'.
`sensitivity`	`str`	no	'public'/'internal'/'confidential'.

`ParserInput`¶

A raw document handed to a Parser: source lineage plus undecoded content.

This is the input side of the Parse seam (ADR-0019). Source produces a ParserInput per source document — Provenance and Classification stamped, but content still the raw bytes (or already-decoded text) exactly as read from Storage, NOT yet turned into markdown. The Parser owns decoding and the turn into a DocumentRecord, so the seam fits real parsers (which consume PDF/image/office BYTES), not just the S1 plain-text passthrough.

config carries per-input Parser hints (e.g. a forced encoding or an OCR language) so a Source or the Runner can steer a Parser without a new contract.

content bytes are serialized to JSON as base64 (ser_json_bytes / val_json_bytes), not as a UTF-8 string. This is a serialization-durability invariant, not a field change: a raw source document can be arbitrary binary (a PDF, an office file, a ZIP archive with a deflate stream), and Pydantic's default bytes-as-UTF-8 JSON encoding raises on any byte sequence that is not valid UTF-8 — which would tear the Runner's checkpoint the moment a genuinely binary file (e.g. a Screened ZIP) is threaded through the Source/Intake seam. base64 round-trips any bytes losslessly (model_validate_json yields the identical bytes), so checkpoint/resume survives the messy, dangerous corpus the demo is built to ingest. The model fields are unchanged, so the inter-Stage contract (and its schema_version) is unchanged — only the on-disk JSON encoding of already-binary content is made lossless.

Field	Type	Required	Description
`schema_version`	`int`	no
`provenance`	`Provenance`	yes
`classification`	`Classification`	yes
`content`	`bytes \\| str`	yes	Raw, undecoded source content — bytes for binary formats, str if text.
`config`	`dict[str, Any]`	no	Per-input Parser hints (encoding, OCR lang, …).
`page_map_sidecar`	`bytes \\| None`	no	Raw bytes of the document's PAGE_MAP_SIDECAR_SUFFIX file, if the Source found one beside it. Undecoded and uninterpreted, exactly like 'content' — the Source knows Storage and can find the file; the Parser owns turning it into a PageMap (ADR-0019/0060). None means no sidecar was present, which is not an error here: what a Parser does about it is the Parser's disposition to record.

`DocumentRecord`¶

A whole-document record: the parsed Parse output.

content is the assembled markdown text; media_type records how to interpret it. page_map (present when disposition == PARSED) records the source page boundaries so any downstream offset resolves back to its original page. disposition/error capture a graceful parse failure — a corrupt file yields a PARSE_ERROR record, never an exception that aborts the run.

Field	Type	Required	Description
`schema_version`	`int`	no
`record_id`	`str`	yes
`provenance`	`Provenance`	yes
`classification`	`Classification`	yes
`content`	`str`	no
`media_type`	`str`	no
`page_map`	`PageMap \\| None`	no
`disposition`	`Disposition`	no
`error`	`str \\| None`	no	Human-readable failure reason when disposition is PARSE_ERROR.
`induced_labels`	`InducedLabels \\| None`	no	Per-document induced label schema (schema_induction Stage), so the per-document Redaction Stage UNIONs the induced pii_types onto its config labels. None (default) ⇒ the document is unchanged and Redaction uses its config labels exactly as before.

`ChunkRecord`¶

One retrieval-sized piece of a parsed document (CONTEXT Chunk).

Chunk splits a DocumentRecord's markdown into retrieval-sized pieces while preserving offsets, page alignment, Provenance and Classification losslessly. Every chunk carries:

char_start/char_end on its Provenance — the half-open span the chunk occupies in the parent document's assembled markdown, so any chunk resolves back to its exact source offsets (round-trip: chunk → offset → original page via PageOffsetIndex);
page_start/page_end on its Provenance — the source page(s) the chunk spans, resolved from the parent's page map;
the parent's Classification, inherited unchanged;
risk_markers — any RiskMarker Content Screening attached, which survive into the corpus so RAG consumers can exclude the chunk.

document_record_id links the chunk back to its parent document; index is the chunk's ordinal within that document (0-based, contiguous).

page_slice carries this chunk's own page spans so a downstream Stage that finds a sub-chunk offset (an entity mention) can resolve that offset's own source page — not merely inherit the chunk's whole [page_start, page_end] range. Without it, a mention on a later page of a multi-page chunk would cite the chunk's first page.

Since v19 the chunk is SELF-DESCRIBING (PageSlice): it holds exactly the document page spans overlapping its own [char_start, char_end) — one for a chunk inside a page, two for a chunk straddling a page break — in the document's original coordinates. v18 instead carried the whole document map on the document's FIRST chunk and had every Stage adopt it from the stream, which made resolution depend on that one record being seen first and surviving every filtering Stage; when it was not, resolution silently degraded to the chunk's inherited page range — a plausible-looking WRONG page. A slice cannot express that failure: there is no shared state and no ordering to depend on.

Resolve through PageIndexResolver rather than reading the field: the resolver owns the half-open end-of-span clamp and the drift diagnostics. A None here means the producing Chunker supplied no page spans at all — resolution raises PageSliceMissingError rather than guessing (ADR-0034).

offset_map carries the stripped-content→original-markdown offset map (OffsetMap) so a downstream Stage can recover a sub-chunk offset's TRUE original position. content is markup-stripped, so char_start + local is only a lower bound on a mention's real offset once markup was stripped before it; the map corrects that (ADR-0031). Optional and additive — absent (pre-v11 chunk, or no map), downstream degrades to the char_start + local shift.

Field	Type	Required	Description
`schema_version`	`int`	no
`record_id`	`str`	yes
`provenance`	`Provenance`	yes
`classification`	`Classification`	yes
`document_record_id`	`str`	yes	Parent DocumentRecord id.
`index`	`int`	yes	0-based chunk ordinal within the document.
`content`	`str`	no
`media_type`	`str`	no
`token_count`	`int`	yes	Budgeted token count for the chunk text.
`page_slice`	`PageSlice \\| None`	no	THIS chunk's own page spans — exactly the document page spans overlapping [char_start, char_end), in document coordinates (v19). Self-describing: it resolves a sub-chunk offset in any order, with any other chunk missing. Read it through latence_core.page_index.PageIndexResolver, never directly.
`offset_map`	`OffsetMap \\| None`	no	Stripped-content→original-markdown offset map, so a sub-chunk offset (an entity mention) resolves to its TRUE original offset, not the lower bound char_start+local.
`risk_markers`	`list[RiskMarker]`	no	Content-Screening flags that propagate into the corpus.
`induced_labels`	`InducedLabels \\| None`	no	Per-document induced label schema the OPTIONAL schema_induction Stage attached, so a downstream extractor UNIONs these types onto its config labels (like risk_markers). None (default) ⇒ the chunk is unchanged and every extractor uses its config labels.
`context_header`	`ContextHeader \\| None`	no	The KG projected back onto this chunk (canonical entities + top-k neighbor triples) the OPTIONAL context_enrichment Stage attached, which the Export prepends to the EMBEDDING input ONLY — the stored content is byte-unchanged (W13, ADR-0039). None (default) ⇒ the chunk is unchanged and the Export prepends nothing (byte-identical to a run without the Stage), mirroring the risk_markers/induced_labels precedent.
`masked_content`	`str \\| None`	no	The PII-handled variant of this chunk's text the OPTIONAL Redaction Stage produced (W16, ADR-0042): the chunk text with every non-TAG PII span replaced by its placeholder. The clean `content` stays UNMASKED (so extraction still sees full entities); the Export materializes `masked_content` for the PII-safe RAG corpus. None (default) ⇒ no Redaction Stage ran, and the Export falls back to `content` (byte-identical to a run without the Stage), mirroring the context_header precedent. Redaction runs per CHUNK (each ≤ the model window) so no PII is truncated — the fix for the document-level 768-token truncation leak (ADR-0042).
`pii_spans`	`list[PIISpan]`	no	The chunk-local PII spans the Redaction Stage detected (W16, ADR-0042): each span's `char_start`/`char_end` are offsets into THIS chunk's `content` (the coordinate system `masked_content` is masked in), while `page_start`/`page_end` resolve to the original source page through the chunk's offset_map+page_slice. Counts-only: a span never stores the raw PII value (S6 AC). Empty (default) ⇒ no Redaction Stage ran, or a clean chunk.
`redaction_disabled_for_sensitive`	`bool`	no	True when this chunk's Classification.sensitivity is in the Redactor's sensitive set (confidential/restricted/…) yet the applied policy ran NO detectors (skip / empty types / empty detector set) — so an empty `pii_spans` here means 'the control was OFF', NOT 'no PII found' (the H-C1 §4 no-op floor carried per chunk, W16/ADR-0042).

`EntityMention`¶

One typed entity mention found in a chunk's text (CONTEXT Entity Extraction).

Entity Extraction finds typed entity mentions in chunked text (zero-shot NER): every mention is an inter-Stage Record — it carries Provenance and Classification like any other — so it can be threaded through the DAG and exported. A mention records:

label — the entity type it was tagged as. The label set is supplied per Pipeline (zero-shot); no label set is hardcoded in core.
text — the exact surface string of the mention as it appears in the parent document's assembled markdown.
confidence — the extractor's 0..1 score for the mention.
char_start/char_end on its Provenance — the half-open span the mention occupies in the parent document's assembled markdown (not in the chunk-local text), so a mention resolves back to its exact source offsets and, through the parent's page map, to its original source page(s) (the S4 round-trip: mention → offset → original page, via PageOffsetIndex).
page_start/page_end on its Provenance — the source page(s) the mention span falls on, resolved from the parent's page map.
the parent chunk's Classification, inherited unchanged.

chunk_record_id/document_record_id link the mention back to the chunk it was found in and to the document that chunk came from; index is the mention's ordinal within its chunk (0-based).

Field	Type	Required	Description
`schema_version`	`int`	no
`record_id`	`str`	yes
`provenance`	`Provenance`	yes
`classification`	`Classification`	yes
`document_record_id`	`str`	yes	Parent DocumentRecord id.
`chunk_record_id`	`str`	yes	Parent ChunkRecord id.
`index`	`int`	yes	0-based mention ordinal within the chunk.
`label`	`str`	yes	Zero-shot entity type (per-Pipeline label).
`text`	`str`	yes	The mention's exact surface string.
`confidence`	`float`	yes	Extractor confidence 0..1.
`raw_label`	`str \\| None`	no	The pre-canonicalization induced type, when a Type-Consolidation Stage ran.

`RelationMention`¶

One typed relation between two entity mentions (CONTEXT Relation Extraction).

Relation Extraction finds typed relations between entity mentions within a document. A relation is an inter-Stage Record — it carries Provenance and Classification like EntityMention — so it threads through the DAG, is checkpointed/resumed uniformly, and is exported to the corpus directly. A relation records:

label — the relation type (e.g. works_for, located_in). The relation label set is supplied per Pipeline (zero-shot / prompt-driven); no label set is hardcoded in core, mirroring Entity Extraction.
head_mention_id/tail_mention_id — the record_id of the head (subject) and tail (object) EntityMention the relation connects. The relation is directed head → tail.
head_text/tail_text — the head/tail surface strings, carried denormalised so a consumer (and Graph Assembly's Evidence) reads the relation without re-joining to the mention set.
confidence — the extractor's 0..1 score for the relation.
char_start/char_end on its Provenance — the half-open span in the parent document's assembled markdown that covers both endpoints (from the earlier mention's start to the later mention's end), so the relation resolves back to source offsets and, through the parent's page map, to its source page(s). The page span is resolved the same way (mention → offset → page).
the parent document's Classification, inherited unchanged.

document_record_id links the relation back to the document its two mentions belong to; index is the relation's ordinal within that document (0-based).

The head and tail must be distinct mentions (no self-relation), and the char span must be present and well-ordered — a relation with no span or an inverted span fails validation, so a Provider that forgets to resolve the endpoint offsets cannot silently emit an un-resolvable relation.

Field	Type	Required	Description
`schema_version`	`int`	no
`record_id`	`str`	yes
`provenance`	`Provenance`	yes
`classification`	`Classification`	yes
`document_record_id`	`str`	yes	Parent DocumentRecord id.
`index`	`int`	yes	0-based relation ordinal within the document.
`label`	`str`	yes	Zero-shot relation type (per-Pipeline label).
`head_mention_id`	`str`	yes	record_id of the head mention.
`tail_mention_id`	`str`	yes	record_id of the tail mention.
`head_text`	`str`	yes	Head mention surface string.
`tail_text`	`str`	yes	Tail mention surface string.
`confidence`	`float`	yes	Extractor confidence 0..1.
`raw_label`	`str \\| None`	no	The pre-canonicalization induced type, when a Type-Consolidation Stage ran.

`FeatureRecord`¶

A corpus-level Profiling result: a typed FeatureSet at one scope (CONTEXT Profiling).

Profiling computes statistical and quality features over the corpus. A FeatureRecord is an inter-Stage Record — it carries Provenance and Classification like every other carrier — so it threads through the DAG, is checkpointed/resumed uniformly, and exports directly. Exactly one of document/corpus is populated, selected by scope:

scope == DOCUMENT: document holds the per-document DocumentFeatures and document_record_id links back to the profiled DocumentRecord; its Provenance is that document's lineage, its Classification inherited unchanged. corpus is None.
scope == CORPUS: corpus holds the cross-document CorpusFeatures; document_record_id is None and its Provenance is the synthetic corpus-scope lineage (the whole run is its source). document is None.

The scope/payload consistency is contract-enforced, so a Provider cannot emit a document-scope record with corpus features (or vice-versa) or leave both empty.

Field	Type	Required	Description
`schema_version`	`int`	no
`record_id`	`str`	yes
`provenance`	`Provenance`	yes
`classification`	`Classification`	yes
`scope`	`FeatureScope`	yes
`document_record_id`	`str \\| None`	no	Parent DocumentRecord id (DOCUMENT scope only).
`document`	`DocumentFeatures \\| None`	no
`corpus`	`CorpusFeatures \\| None`	no

`DisambiguationRecord`¶

A corpus-level Disambiguation result: one canonical entity OR one normalized relation.

The scoped inter-Stage carrier for the Disambiguation Stage, mirroring FeatureRecord's scope split (ADR-0025/0026). Exactly one of entity/relation is populated, selected by scope:

scope == ENTITY: entity holds the CanonicalEntity; relation is None. Its Provenance is the canonical mention's page-accurate lineage and its Classification is inherited from that mention.
scope == RELATION: relation holds the NormalizedRelation; entity is None. Its Provenance/Classification are the source relation's.

The scope/payload consistency is contract-enforced, so a Provider cannot emit an entity-scope record with a relation payload (or leave both empty). Records are emitted deterministically (all ENTITY records in canonical-text order, then all RELATION records) so a seeded run is byte-identical (Baseline bar).

Field	Type	Required
`schema_version`	`int`	no
`record_id`	`str`	yes
`provenance`	`Provenance`	yes
`classification`	`Classification`	yes
`scope`	`DisambiguationScope`	yes
`entity`	`CanonicalEntity \\| None`	no
`relation`	`NormalizedRelation \\| None`	no

`GraphRecord`¶

A corpus-level Graph-Assembly result: one graph node OR one graph edge.

The scoped inter-Stage carrier for the Graph Assembly Stage, mirroring DisambiguationRecord's scope split (ADR-0026). Exactly one of node/edge is populated, selected by scope:

scope == NODE: node holds the GraphNode; edge/hyperedge are None. Its Provenance/Classification are the source canonical entity's (page-accurate).
scope == EDGE: edge holds the GraphEdge; node/hyperedge are None. Its Provenance/Classification are the source normalized relation's.
scope == HYPEREDGE: hyperedge holds the GraphHyperedge (ADR-0057); node/edge are None. Its Provenance is the primary span's, in original coordinates.

The scope/payload consistency is contract-enforced, so a Provider cannot emit a node-scope record with an edge payload (or leave both empty). Records are emitted deterministically (all NODE records in node-id order, then all EDGE records in edge-id order, then any HYPEREDGE records in hyperedge-id order), so a seeded run is byte-identical (Baseline bar).

Field	Type	Required
`schema_version`	`int`	no
`record_id`	`str`	yes
`provenance`	`Provenance`	yes
`classification`	`Classification`	yes
`scope`	`GraphScope`	yes
`node`	`GraphNode \\| None`	no
`edge`	`GraphEdge \\| None`	no
`hyperedge`	`GraphHyperedge \\| None`	no

Stage contracts — the replacement contract for every Capability¶

Replacing a Provider — the five steps¶

How to read an entry¶

Stage Capabilities¶

source — Source¶

Protocol Source¶

Optional refinement — DocumentEnumerator¶

intake_screening — IntakeScreener¶

Protocol IntakeScreener¶

parse — Parser¶

Protocol Parser¶

chunk — Chunker¶

Protocol Chunker¶

content_screening — ContentScreener¶

Protocol ContentScreener¶

schema_induction — LabelInducer¶

Protocol LabelInducer¶

entity_extraction — EntityExtractor¶

Protocol EntityExtractor¶

relation_extraction — RelationExtractor¶

Protocol RelationExtractor¶

fused_entity_relation — FusedEntityRelationExtractor¶

Protocol FusedEntityRelationExtractor¶

redaction — PIIDetector¶

Protocol PIIDetector¶

profiling — Profiler¶

Protocol Profiler¶

type_consolidation — TypeConsolidator¶

Protocol TypeConsolidator¶

disambiguation — Disambiguator¶

Protocol Disambiguator¶

graph_assembly — GraphAssembler¶

Protocol GraphAssembler¶

graph_completion — GraphCompleter¶

Protocol GraphCompleter¶

context_enrichment — ContextEnricher¶

Protocol ContextEnricher¶

embedding — Embedder¶

Protocol Embedder¶

export — Export¶

Protocol Export¶

Signal seams (non-Stage Capabilities)¶

sparse — SparseEmbedder¶

Protocol SparseEmbedder¶

Optional refinement — ComposingSparseEmbedder¶

multivector — MultiVectorEmbedder¶

Protocol MultiVectorEmbedder¶

fde — FdeConverter¶

Protocol FdeConverter¶

tokenizer — Tokenizer¶

Protocol Tokenizer¶

graph_features — GraphFeatureComputer¶

Protocol GraphFeatureComputer¶

delta — DeltaProcessor¶

Protocol DeltaProcessor¶

Protocol result types¶

ScreeningOutcome¶

FusedExtraction¶

TypeConsolidation¶

DeltaOutcome¶

GraphFeatures¶

SparseVector¶

SparseSignals¶

Record carriers¶

Record¶

Provenance¶

Classification¶

ParserInput¶

DocumentRecord¶

ChunkRecord¶

EntityMention¶

RelationMention¶

FeatureRecord¶

DisambiguationRecord¶

GraphRecord¶

`source` — Source¶

Protocol `Source`¶

Optional refinement — `DocumentEnumerator`¶

`intake_screening` — IntakeScreener¶

Protocol `IntakeScreener`¶

`parse` — Parser¶

Protocol `Parser`¶

`chunk` — Chunker¶

Protocol `Chunker`¶

`content_screening` — ContentScreener¶

Protocol `ContentScreener`¶

`schema_induction` — LabelInducer¶

Protocol `LabelInducer`¶

`entity_extraction` — EntityExtractor¶

Protocol `EntityExtractor`¶

`relation_extraction` — RelationExtractor¶

Protocol `RelationExtractor`¶

`fused_entity_relation` — FusedEntityRelationExtractor¶

Protocol `FusedEntityRelationExtractor`¶

`redaction` — PIIDetector¶

Protocol `PIIDetector`¶

`profiling` — Profiler¶

Protocol `Profiler`¶

`type_consolidation` — TypeConsolidator¶

Protocol `TypeConsolidator`¶

`disambiguation` — Disambiguator¶

Protocol `Disambiguator`¶

`graph_assembly` — GraphAssembler¶

Protocol `GraphAssembler`¶

`graph_completion` — GraphCompleter¶

Protocol `GraphCompleter`¶

`context_enrichment` — ContextEnricher¶

Protocol `ContextEnricher`¶

`embedding` — Embedder¶

Protocol `Embedder`¶

`export` — Export¶

Protocol `Export`¶

`sparse` — SparseEmbedder¶

Protocol `SparseEmbedder`¶

Optional refinement — `ComposingSparseEmbedder`¶

`multivector` — MultiVectorEmbedder¶

Protocol `MultiVectorEmbedder`¶

`fde` — FdeConverter¶

Protocol `FdeConverter`¶

`tokenizer` — Tokenizer¶

Protocol `Tokenizer`¶

`graph_features` — GraphFeatureComputer¶

Protocol `GraphFeatureComputer`¶

`delta` — DeltaProcessor¶

Protocol `DeltaProcessor`¶

`ScreeningOutcome`¶

`FusedExtraction`¶

`TypeConsolidation`¶

`DeltaOutcome`¶

`GraphFeatures`¶

`SparseVector`¶

`SparseSignals`¶

`Record`¶

`Provenance`¶

`Classification`¶

`ParserInput`¶

`DocumentRecord`¶

`ChunkRecord`¶

`EntityMention`¶

`RelationMention`¶

`FeatureRecord`¶

`DisambiguationRecord`¶

`GraphRecord`¶