Data contracts

The single Pydantic schema every inter-Stage Record shares, serialized to both JSONL and Parquet/Arrow (ADR-0006). Provenance and Classification are the cross-cutting carriers every record holds.

contracts

Versioned data contracts shared across every Stage boundary.

One schema, dual serialization (ADR-0006): these Pydantic v2 models are the single canonical definition of inter-Stage data. Every :class:Record carries both :class:Provenance (immutable source lineage) and :class:Classification (descriptive content attributes); validation fails if either is missing, so a record can never silently lose its source or attributes at a Stage boundary.

Provenance

Bases: BaseModel

The immutable chain from a record back to its source.

Required fields identify the source document unambiguously; the optional offset fields locate a derived record within it (populated by Parse/Chunk).

Classification

Bases: BaseModel

Descriptive attributes of a document's content.

Attached at ingest / after Parse and inherited by every downstream record.

Record

Bases: BaseModel

Base inter-Stage record: always carries Provenance and Classification.

Both are required with no default, so constructing a record without them raises pydantic.ValidationError — the enforced Stage-boundary check.

PageMapOrigin

Bases: StrEnum

Where a :class:PageMap's boundaries came from — measurement or convention.

Before this existed, "this document really is one page" and "I have no idea how this document is paginated" were the SAME value: a single span covering everything, page 1. Anything downstream that cited a page therefore cited page 1 with full confidence for an entire OCR'd corpus (the ADR-0060 defect). Recording the origin does not recover the boundaries — nothing can, from an assembled blob — it makes their ABSENCE a value a consumer can see, filter on, and refuse.

PARSER — the Parser segmented the source itself (a PDF reader walking pages, an OCR Provider that ran per page). The boundaries are measured.

SIDECAR — the boundaries were supplied out-of-band, in a :data:PAGE_MAP_SIDECAR_SUFFIX file, by whoever produced the markdown. Measured by the producer and cross-checked against the decoded text on the way in, so as trustworthy as PARSER — but distinguishable, because the trust rests on a different artifact.

ASSUMED_SINGLE_PAGE — there was NO page structure available: a .txt note, or markdown whose pagination was lost upstream. The single span is a convention so that offset→page resolution has one code path, not a claim that the document has one page. A consumer that needs real page citation must treat this as unknown, not as page 1.

PageSpan

Bases: BaseModel

One page's half-open [char_start, char_end) span in the assembled text.

PageMap

Bases: BaseModel

The page-boundary map carried on a parsed :class:DocumentRecord.

pages are contiguous, non-overlapping, ordered spans covering the whole assembled text; total_chars is the assembled length (the end of the last page). A single-page document (plain text) has exactly one span — the map is always present after a successful Parse, never optional, so downstream offset→page resolution has one code path. The live resolver over this map is :class:latence_core.pagemap.PageOffsetIndex.

origin says whether those boundaries were MEASURED or ASSUMED (v21/ADR-0060). One span with origin=PARSER is a one-page document; one span with origin=ASSUMED_SINGLE_PAGE is a document whose pagination is unknown. They used to be indistinguishable, which is how an entire OCR'd corpus came to cite page 1.

from_page_texts classmethod

from_page_texts(
    page_texts: list[str],
    separator: str = "\n\n",
    *,
    origin: PageMapOrigin = PageMapOrigin.PARSER
) -> PageMapAssembly

Assemble per-page text into one markdown string plus its page map.

The Parser-side constructor: hand it the per-page decoded text in page order and it returns both the concatenated markdown and the :class:PageMap recording where each page landed — the two are built together so the boundaries are exact by construction, not reverse-engineered.

A single-page (or zero-page, coerced to one empty page) document is the plain-text case: one span covering everything. A caller in that position — a Parser handed a page-less blob — passes origin=PageMapOrigin.ASSUMED_SINGLE_PAGE so the record says so; a Parser that really walked the source's pages keeps the PARSER default.

Source code in packages/latence-core/src/latence_core/contracts.py

@classmethod
def from_page_texts(
    cls,
    page_texts: list[str],
    separator: str = "\n\n",
    *,
    origin: PageMapOrigin = PageMapOrigin.PARSER,
) -> PageMapAssembly:
    """Assemble per-page text into one markdown string plus its page map.

    The Parser-side constructor: hand it the per-page decoded text in page
    order and it returns both the concatenated markdown and the
    :class:`PageMap` recording where each page landed — the two are built
    together so the boundaries are exact by construction, not reverse-engineered.

    A single-page (or zero-page, coerced to one empty page) document is the
    plain-text case: one span covering everything. A caller in that position —
    a Parser handed a page-less blob — passes
    ``origin=PageMapOrigin.ASSUMED_SINGLE_PAGE`` so the record says so; a Parser
    that really walked the source's pages keeps the ``PARSER`` default.
    """
    if not page_texts:
        page_texts = [""]
    spans: list[PageSpan] = []
    parts: list[str] = []
    cursor = 0
    sep_len = len(separator)
    last = len(page_texts) - 1
    for i, text in enumerate(page_texts):
        start = cursor
        parts.append(text)
        cursor += len(text)
        # The inter-page separator counts toward the *preceding* page's span so
        # the assembled offsets stay contiguous with no un-attributed gap.
        if i != last:
            parts.append(separator)
            cursor += sep_len
        spans.append(PageSpan(page_number=i + 1, char_start=start, char_end=cursor))
    assembled = "".join(parts)
    page_map = cls(pages=spans, total_chars=len(assembled), origin=origin)
    return PageMapAssembly(text=assembled, page_map=page_map)

from_sidecar_json classmethod

from_sidecar_json(
    data: bytes | str, *, text_length: int
) -> PageMap

Read a :data:PAGE_MAP_SIDECAR_SUFFIX sidecar into a validated SIDECAR map.

The consumer half of the out-of-band handoff (ADR-0060). data is the sidecar file exactly as it was read — its own origin, if it carries one, is IGNORED and overwritten with :attr:PageMapOrigin.SIDECAR, because a file cannot vouch for how the boundaries in it were obtained; what a consumer can honestly record is that they arrived out of band.

text_length is the length of the DECODED document text the map is supposed to describe, and the check that makes the sidecar trustworthy: a sidecar whose total_chars disagrees is stale (the markdown was regenerated, re-encoded, or hand-edited), and its spans would resolve real offsets to confidently wrong pages. That raises :class:~latence_core.errors.PageMapSidecarError — which a Parser turns into a PARSE_ERROR record — rather than returning a map that looks fine.

Every other malformation (unreadable bytes, invalid JSON, a non-object payload, spans that are not contiguous, a schema version from the future) raises the same error, so a caller has exactly one failure to handle.

Source code in packages/latence-core/src/latence_core/contracts.py

@classmethod
def from_sidecar_json(cls, data: bytes | str, *, text_length: int) -> PageMap:
    """Read a :data:`PAGE_MAP_SIDECAR_SUFFIX` sidecar into a validated ``SIDECAR`` map.

    The consumer half of the out-of-band handoff (ADR-0060). ``data`` is the sidecar file
    exactly as it was read — its own ``origin``, if it carries one, is IGNORED and
    overwritten with :attr:`PageMapOrigin.SIDECAR`, because a file cannot vouch for how the
    boundaries in it were obtained; what a consumer can honestly record is that they arrived
    out of band.

    ``text_length`` is the length of the DECODED document text the map is supposed to
    describe, and the check that makes the sidecar trustworthy: a sidecar whose
    ``total_chars`` disagrees is stale (the markdown was regenerated, re-encoded, or
    hand-edited), and its spans would resolve real offsets to confidently wrong pages. That
    raises :class:`~latence_core.errors.PageMapSidecarError` — which a Parser turns into a
    ``PARSE_ERROR`` record — rather than returning a map that looks fine.

    Every other malformation (unreadable bytes, invalid JSON, a non-object payload, spans
    that are not contiguous, a schema version from the future) raises the same error, so a
    caller has exactly one failure to handle.
    """
    if isinstance(data, bytes):
        try:
            text = data.decode("utf-8")
        except UnicodeDecodeError as exc:
            raise PageMapSidecarError(f"sidecar is not valid UTF-8: {exc}") from exc
    else:
        text = data
    try:
        payload = json.loads(text)
    except ValueError as exc:
        raise PageMapSidecarError(f"sidecar is not valid JSON: {exc}") from exc
    if not isinstance(payload, dict):
        raise PageMapSidecarError(
            f"sidecar must be a JSON object, got {type(payload).__name__}"
        )
    fields = {key: value for key, value in payload.items() if key != "origin"}
    declared = fields.get("schema_version")
    if isinstance(declared, int) and declared > SCHEMA_VERSION:
        raise PageMapSidecarError(
            f"sidecar declares schema_version {declared}, newer than this build's "
            f"{SCHEMA_VERSION} — refusing to guess what changed"
        )
    try:
        page_map = cls.model_validate({**fields, "origin": PageMapOrigin.SIDECAR})
    except ValidationError as exc:
        raise PageMapSidecarError(f"sidecar is not a valid PageMap: {exc}") from exc
    if page_map.total_chars != text_length:
        raise PageMapSidecarError(
            f"sidecar describes {page_map.total_chars} characters but the decoded document "
            f"is {text_length} — the sidecar is stale, so its page spans would resolve "
            f"offsets to the wrong pages"
        )
    return page_map

PageMapAssembly

Bases: BaseModel

The assembled markdown plus its :class:PageMap, built together by Parse.

PageSlice

Bases: BaseModel

The page spans overlapping ONE chunk — that chunk's own, self-describing page map.

A :class:PageMap covers a whole document and therefore starts at page 1 and at character 0. A slice is a window onto one: exactly the :class:PageSpan s that overlap [chunk.char_start, chunk.char_end), in the document's ORIGINAL character coordinates and with their original page_number s. A chunk that sits inside one page carries one span; a chunk straddling a page break carries two; the pathological case (a chunk spanning many tiny pages) carries a handful. That is the whole of what a chunk needs, because the only question asked of it is sub-chunk: which page does an offset INSIDE this chunk fall on.

Why this exists (v19). The map used to be document-scoped state carried on ONE record of the chunk stream — the document's first chunk (v18) — and adopted by a resolver from whichever record it happened to see first. That made page resolution depend on two things that are not properties of the chunk being resolved: that the map-carrying record was observed FIRST (order dependence), and that it SURVIVED every filtering Stage (drop dependence). Both failure modes returned a plausible-looking WRONG page. A slice removes the dependency by removing the shared state: every chunk answers for itself, in any order, with any other chunk missing.

The payload win the v18 carriage existed for is kept: ~2 spans (~60 bytes) per chunk against the ~30 KB whole-document map a 519-page document would otherwise duplicate.

Coordinates are NOT rebased to the chunk. char_start/char_end stay in document space so resolution arithmetic is identical to the document map's and the provenance a record cites is the document's own (ADR-0021/0031).

for_span classmethod

for_span(
    page_map: PageMap, char_start: int, char_end: int
) -> PageSlice

The slice of page_map overlapping the half-open [char_start, char_end).

char_end is exclusive, so the last page included is the one holding the last included character — a chunk ending exactly on a page boundary does NOT drag in the page that starts there. A zero-length or inverted span still yields the one page containing char_start, so a slice is never empty.

Source code in packages/latence-core/src/latence_core/contracts.py

@classmethod
def for_span(cls, page_map: PageMap, char_start: int, char_end: int) -> PageSlice:
    """The slice of ``page_map`` overlapping the half-open ``[char_start, char_end)``.

    ``char_end`` is exclusive, so the last page included is the one holding the last
    included character — a chunk ending exactly on a page boundary does NOT drag in the
    page that starts there. A zero-length or inverted span still yields the one page
    containing ``char_start``, so a slice is never empty.
    """
    last_included = max(char_end - 1, char_start)
    # Cut by INDEX with the same ``bisect_right - 1`` the live index resolves with
    # (:meth:`latence_core.pagemap.PageOffsetIndex.page_for_offset`), so the slice
    # provably contains the page the full document map would have answered for every
    # offset in the span — and, being an index range, is contiguous and consecutively
    # numbered even when the document contains a zero-length page.
    starts = [span.char_start for span in page_map.pages]
    last = len(page_map.pages) - 1
    lo = min(max(bisect.bisect_right(starts, char_start) - 1, 0), last)
    hi = min(max(bisect.bisect_right(starts, last_included) - 1, lo), last)
    return cls(
        pages=page_map.pages[lo : hi + 1],
        total_chars=page_map.total_chars,
        # The slice inherits the map's honesty, not just its numbers: a chunk cut from an
        # assumed single-page map must not read, downstream, like a chunk cut from a real
        # per-page OCR run.
        origin=page_map.origin,
    )

Disposition

Bases: StrEnum

A parsed record's outcome disposition.

PARSED is the healthy path. PARSE_ERROR marks a document the Parser could not decode (a corrupt PDF, an unsupported/spoofed format): the record is still emitted — carrying its Provenance and the failure reason — so the run does not crash and the failure is auditable in the Quality Report, but it carries no usable content and downstream Stages skip it.

DocumentRecord

Bases: Record

A whole-document record: the parsed Parse output.

content is the assembled markdown text; media_type records how to interpret it. page_map (present when disposition == PARSED) records the source page boundaries so any downstream offset resolves back to its original page. disposition/error capture a graceful parse failure — a corrupt file yields a PARSE_ERROR record, never an exception that aborts the run.

size_after_parse property

size_after_parse: int

Assembled-markdown size in bytes — the Provenance 'size-after-parse' (S2).

A property of the RECORD, not of any Parser: it reads content and nothing else, so every Parser that produced one of these answers it identically. It used to be six byte-identical size_after_parse(record) helpers, one per Parser package, each docstring claiming to "mirror" the others — six copies of the same one-liner is six chances for the signal the Runner sums into ParseQuality.bytes_after_parse to stop meaning the same thing across Providers. Those helpers survive as one-line delegations to this property, because each package publishes its own in __all__.

UTF-8 because that is the encoding the Parse seam assembles markdown in; the count is of ENCODED BYTES, not characters, so a document of accented or CJK text reports the size a sink actually stores.

ParserInput

Bases: BaseModel

A raw document handed to a Parser: source lineage plus undecoded content.

This is the input side of the Parse seam (ADR-0019). Source produces a ParserInput per source document — Provenance and Classification stamped, but content still the raw bytes (or already-decoded text) exactly as read from Storage, NOT yet turned into markdown. The Parser owns decoding and the turn into a :class:DocumentRecord, so the seam fits real parsers (which consume PDF/image/office BYTES), not just the S1 plain-text passthrough.

config carries per-input Parser hints (e.g. a forced encoding or an OCR language) so a Source or the Runner can steer a Parser without a new contract.

content bytes are serialized to JSON as base64 (ser_json_bytes / val_json_bytes), not as a UTF-8 string. This is a serialization-durability invariant, not a field change: a raw source document can be arbitrary binary (a PDF, an office file, a ZIP archive with a deflate stream), and Pydantic's default bytes-as-UTF-8 JSON encoding raises on any byte sequence that is not valid UTF-8 — which would tear the Runner's checkpoint the moment a genuinely binary file (e.g. a Screened ZIP) is threaded through the Source/Intake seam. base64 round-trips any bytes losslessly (model_validate_json yields the identical bytes), so checkpoint/resume survives the messy, dangerous corpus the demo is built to ingest. The model fields are unchanged, so the inter-Stage contract (and its schema_version) is unchanged — only the on-disk JSON encoding of already-binary content is made lossless.

ScreeningCheckpoint

Bases: StrEnum

Which of Screening's two checkpoints produced a finding (CONTEXT glossary).

INTAKE runs before Parse over the raw document (malware, zip bombs, file-type spoofing, oversized/corrupt files). CONTENT runs after Chunk over the retrieval-sized text (prompt injection, harmful content, sensitivity escalation).

ScreeningVerdict

Bases: StrEnum

What Screening decided about a record.

PASS — nothing found; the record proceeds untouched. FLAG — a risk was found but the record still proceeds, carrying a :class:RiskMarker so a RAG consumer can exclude it. QUARANTINE — the record is dangerous and is removed from every downstream Stage and Export (retained and inspectable with the reason, CONTEXT Quarantine).

RiskMarker

Bases: BaseModel

A risk flag Content Screening attaches to a record so it propagates.

The marker survives into the corpus (it is a field on the record contract, not an out-of-band note) so a downstream RAG consumer can filter on it — the S3 acceptance criterion "a risk marker that survives into the corpus so RAG consumers can exclude it".

InducedLabels

Bases: BaseModel

Per-document label schema the schema_induction Stage induced (CONTEXT Classification). W4 restores the "point it at a folder and it discovers its own schema" property: an OPTIONAL :class:~latence_core.capability.LabelInducer Stage (a small OpenAI-compatible LLM with structured JSON output) reads a document and induces the entity / relation / PII types actually evidenced in it, then annotates every record it produced with this carrier so the downstream extraction Stages can UNION these induced types onto their config label floor (config_labels ∪ induced, deduped, order-stable). Like :class:RiskMarker, it is a field on the record contract (not an out-of-band note), so it survives into the corpus and a downstream Stage reads it off the record it already holds — the additive extractor seam.

Every list is already sanitized by the inducing Provider (control chars stripped, length ≤ 64, count ≤ 40, deduped, normalized) — an :class:InducedLabels is trusted, bounded data a zero-shot extractor prompt can consume, never executed. model_id records which LLM induced it (audit); confidences maps a label to the model's 0..1 score for it (auditable, optional — a label with no score is simply absent from the map). A slot the model evidenced no types for is an empty list, not a fabricated one.

v17 adds the fourth slot guard_types (guardrail categories), so every LEARNED label set a downstream Provider prompts with — entity, relation, PII, guardrail — has a per-record slot. Whether a slot's config labels are a droppable RELEVANCE floor or a non-subsettable UNIVERSAL SAFETY floor is the consumer's decision, expressed at the :func:latence_core.induced.effective_labels seam via :class:~latence_core.induced.LabelFloor.

ScreeningFinding

Bases: BaseModel

One Screening decision about one record, recorded for the Quality Report.

Emitted at either checkpoint. A QUARANTINE verdict names the removed document; a FLAG carries the :class:RiskMarker that was propagated. The finding is the audit trail: every Quarantine disposition and reason surfaces in the Quality Report (S3 AC), and a flag records what was marked and why.

QuarantineRecord

Bases: Record

A record Screening removed from the pipeline (CONTEXT Quarantine).

Excluded from every downstream Stage and Export, but retained and inspectable: Provenance is intact, the offending checkpoint/detector/reason are recorded, and the raw byte size is kept so the audit knows what was held back. It never reaches Parse (intake) or the corpus (content). A QuarantineRecord carries no usable content — the point of Quarantine is that the data does not flow, only its audit trail.

OffsetSegment

Bases: BaseModel

One contiguous run mapping stripped-content offsets to original offsets.

Chunk content is the markup-stripped text; markup stripping only ever deletes characters (chunking.strip_markup), so a chunk's surviving characters form contiguous runs of the ORIGINAL parsed markdown. Within a run the map is a constant shift: original offset = doc_start + (local - local_start). A new segment begins at each stripped gap, so the whole stripped→original map is this short list of run breakpoints (typically a handful per chunk), not a per-character table.

local_start is the offset into the chunk's content; doc_start is the original-markdown offset of that same character. The first segment always starts at local_start == 0.

OffsetMap

Bases: BaseModel

Run-length map from a chunk's stripped content to original offsets.

Carried on a :class:ChunkRecord so a downstream Stage that finds a sub-chunk offset (an entity mention's chunk-local match) can recover that offset's TRUE position in the parent document's assembled markdown — not merely the lower bound char_start + local_offset gives once markup was stripped before the mention within its chunk (ADR-0022 corrected, ADR-0031). The live resolver over this map is :class:latence_core.offsetmap.OffsetIndex.

segments are ordered by local_start (strictly increasing, first at 0); length is len(content) — the exclusive upper bound of the local domain, so a half-open local span's original end is resolvable. A markup-free chunk has exactly one segment (0, char_start): the map is then the identity shift and a mention resolves to char_start + local exactly as before.

ContextHeader

Bases: BaseModel

The KG projected back onto a chunk for retrieval coherence (CONTEXT Context Enrichment).

W13 (ADR-0039) restores the connections a chunk needs for retrieval relevance — its canonical entities and their top-k knowledge-graph neighbor triples — that the RAG export otherwise discards by embedding each chunk in isolation. The OPTIONAL context_enrichment Stage builds this compact header per chunk from the run's already-computed :class:EntityMention + :class:GraphRecord streams and the Export prepends it to the EMBEDDING input only: the coherence inflates the vector + a metadata column, never the stored chunk text (the LLM window stays the clean chunk, masked_content/content byte-unchanged).

Like :class:RiskMarker / :class:InducedLabels it is a field on the record contract (not an out-of-band note), so it survives into the corpus and a downstream retriever reads it off the chunk it already holds. It is deliberately COMPACT (the bloat discipline, ADR-0039): compact rendered triples not neighbor prose, a top-k neighbor + confidence cap (the hub-entity guard), and join keys not raw KG text (query-time KG expansion is the retriever's job — ADR-0017, files not a live DB; W13 only emits the keys).

• entities — the chunk's canonical entity names (sorted, deduped), and entity_types the aligned entity type per name (same length + order as entities). entity_confidences is the aligned per-entity 0..1 admission confidence (ADR-0053) — the value the enrichment gate admitted the entity at, carried so a downstream composing SparseEmbedder's confidence_threshold gates weak NER at index time by the identical rule; [] for a header built without it.
• neighbor_triples — the chunk's top KG-neighbor relations rendered A —label→ B, confidence-sorted and CAPPED (max_neighbors_per_entity per entity, max_triples overall) — the anti-bloat + hub-entity guard: a corpus-wide hub entity contributes only its TOP relations, never all of them.
• kg_node_ids — the deterministic content-addressed node ids of the chunk's own entities (the join keys a downstream retriever hybridises/reranks on).
• neighbor_node_ids — the node ids of the neighbor entities the triples reach (the query-time KG-expansion keys), sorted + deduped.
• kg_node_centrality / kg_node_community — the OPTIONAL ADR-0051 graph-feature projection: each entity's index-time centrality score and community id, aligned 1:1 with kg_node_ids (same length + order), so a downstream ranker reads a chunk's own entities' structural importance/cluster off the header it already holds. Both are [] when the graph_features enrichment is OFF (the nodes carry no centrality) — byte-identical to a run without it — and both full-length (aligned with kg_node_ids) when it is ON.
• situating_sentence — OPTIONAL LLM-generated Contextual-Retrieval-style sentence (deterministic=False, a documented follow-on, ADR-0039); None in the deterministic default reference build. model_id records the LLM when that mode is used (audit), None for the deterministic build.

is_empty property

is_empty: bool

True if the header carries no entities and no triples (nothing to prepend).

ChunkRecord

Bases: Record

One retrieval-sized piece of a parsed document (CONTEXT Chunk).

Chunk splits a :class:DocumentRecord's markdown into retrieval-sized pieces while preserving offsets, page alignment, Provenance and Classification losslessly. Every chunk carries:

• char_start/char_end on its Provenance — the half-open span the chunk occupies in the parent document's assembled markdown, so any chunk resolves back to its exact source offsets (round-trip: chunk → offset → original page via :class:~latence_core.pagemap.PageOffsetIndex);
• page_start/page_end on its Provenance — the source page(s) the chunk spans, resolved from the parent's page map;
• the parent's Classification, inherited unchanged;
• risk_markers — any :class:RiskMarker Content Screening attached, which survive into the corpus so RAG consumers can exclude the chunk.

document_record_id links the chunk back to its parent document; index is the chunk's ordinal within that document (0-based, contiguous).

page_slice carries this chunk's own page spans so a downstream Stage that finds a sub-chunk offset (an entity mention) can resolve that offset's own source page — not merely inherit the chunk's whole [page_start, page_end] range. Without it, a mention on a later page of a multi-page chunk would cite the chunk's first page.

Since v19 the chunk is SELF-DESCRIBING (:class:PageSlice): it holds exactly the document page spans overlapping its own [char_start, char_end) — one for a chunk inside a page, two for a chunk straddling a page break — in the document's original coordinates. v18 instead carried the whole document map on the document's FIRST chunk and had every Stage adopt it from the stream, which made resolution depend on that one record being seen first and surviving every filtering Stage; when it was not, resolution silently degraded to the chunk's inherited page range — a plausible-looking WRONG page. A slice cannot express that failure: there is no shared state and no ordering to depend on.

Resolve through :class:~latence_core.page_index.PageIndexResolver rather than reading the field: the resolver owns the half-open end-of-span clamp and the drift diagnostics. A None here means the producing Chunker supplied no page spans at all — resolution raises :class:~latence_core.errors.PageSliceMissingError rather than guessing (ADR-0034).

offset_map carries the stripped-content→original-markdown offset map (:class:OffsetMap) so a downstream Stage can recover a sub-chunk offset's TRUE original position. content is markup-stripped, so char_start + local is only a lower bound on a mention's real offset once markup was stripped before it; the map corrects that (ADR-0031). Optional and additive — absent (pre-v11 chunk, or no map), downstream degrades to the char_start + local shift.

flagged property

flagged: bool

True if Content Screening attached any risk marker to this chunk.

EntityMention

Bases: Record

One typed entity mention found in a chunk's text (CONTEXT Entity Extraction).

Entity Extraction finds typed entity mentions in chunked text (zero-shot NER): every mention is an inter-Stage :class:Record — it carries Provenance and Classification like any other — so it can be threaded through the DAG and exported. A mention records:

• label — the entity type it was tagged as. The label set is supplied per Pipeline (zero-shot); no label set is hardcoded in core.
• text — the exact surface string of the mention as it appears in the parent document's assembled markdown.
• confidence — the extractor's 0..1 score for the mention.
• char_start/char_end on its Provenance — the half-open span the mention occupies in the parent document's assembled markdown (not in the chunk-local text), so a mention resolves back to its exact source offsets and, through the parent's page map, to its original source page(s) (the S4 round-trip: mention → offset → original page, via :class:~latence_core.pagemap.PageOffsetIndex).
• page_start/page_end on its Provenance — the source page(s) the mention span falls on, resolved from the parent's page map.
• the parent chunk's Classification, inherited unchanged.

chunk_record_id/document_record_id link the mention back to the chunk it was found in and to the document that chunk came from; index is the mention's ordinal within its chunk (0-based).

RelationMention

Bases: Record

One typed relation between two entity mentions (CONTEXT Relation Extraction).

Relation Extraction finds typed relations between entity mentions within a document. A relation is an inter-Stage :class:Record — it carries Provenance and Classification like :class:EntityMention — so it threads through the DAG, is checkpointed/resumed uniformly, and is exported to the corpus directly. A relation records:

• label — the relation type (e.g. works_for, located_in). The relation label set is supplied per Pipeline (zero-shot / prompt-driven); no label set is hardcoded in core, mirroring Entity Extraction.
• head_mention_id/tail_mention_id — the record_id of the head (subject) and tail (object) :class:EntityMention the relation connects. The relation is directed head → tail.
• head_text/tail_text — the head/tail surface strings, carried denormalised so a consumer (and Graph Assembly's Evidence) reads the relation without re-joining to the mention set.
• confidence — the extractor's 0..1 score for the relation.
• char_start/char_end on its Provenance — the half-open span in the parent document's assembled markdown that covers both endpoints (from the earlier mention's start to the later mention's end), so the relation resolves back to source offsets and, through the parent's page map, to its source page(s). The page span is resolved the same way (mention → offset → page).
• the parent document's Classification, inherited unchanged.

document_record_id links the relation back to the document its two mentions belong to; index is the relation's ordinal within that document (0-based).

The head and tail must be distinct mentions (no self-relation), and the char span must be present and well-ordered — a relation with no span or an inverted span fails validation, so a Provider that forgets to resolve the endpoint offsets cannot silently emit an un-resolvable relation.

TypeKind

Bases: StrEnum

Which label vocabulary a :class:CanonicalType belongs to.

Entity types and relation types are canonicalized in the SAME pass but never mixed: an entity type organization and a relation type organization of are different vocabularies, and folding one into the other would corrupt the graph. Every lookup is therefore keyed by (kind, label).

CanonicalType

Bases: BaseModel

One canonical type and every raw induced label folded into it (T3).

Per-chunk induction produces type drift — org here, organization there, company in a third chunk — because each chunk induces its own labels. The Type-Consolidation Stage clusters those synonymous surfaces and elects ONE canonical label per cluster; this record is the cluster.

aliases holds every OTHER raw label in the cluster (sorted, canonical excluded), so the alias→canonical map an operator reads on the Quality Report is exactly {alias: canonical for alias in aliases}. strategies names the clustering rungs that produced the folds (normalized/alias/acronym/prefix/head_noun/ embedding), so a suspicious merge is traceable to the rule that made it — the audit the entity resolver's merge_decisions provides on the entity side. mention_count is how many mentions/relations carried a member label, the salience signal an operator sorts by.

labels property

labels: list[str]

Every raw label this cluster covers — the canonical one plus its aliases.

TypeVocabulary

Bases: BaseModel

The corpus's canonical type vocabulary + its alias map (T3) — persistable and mergeable.

The artifact the Type-Consolidation Stage emits: the M canonical types the run's N raw induced types collapsed to, each with the aliases folded into it. It is a plain serializable model (not a :class:Record) because it describes the corpus's SCHEMA, not a span of a document — it has no Provenance to carry. The Runner persists it beside the Stage's checkpoint, which is what makes it survive the run (ADR-0017: files are the source of truth).

Mergeability is a first-class property (designed for the T4 incremental-delta lane): a later run merges its freshly-induced vocabulary INTO the persisted one with :meth:merge, which is deliberately append-don't-churn — an existing canonical label is NEVER renamed by a later run (that would rewrite the type of every node already in the KG); new aliases attach to the existing cluster and genuinely new types are appended.

of_kind

of_kind(kind: TypeKind) -> list[CanonicalType]

The clusters of one kind, in emission order (canonical-label order).

Source code in packages/latence-core/src/latence_core/contracts.py

def of_kind(self, kind: TypeKind) -> list[CanonicalType]:
    """The clusters of one kind, in emission order (canonical-label order)."""
    return [entry for entry in self.types if entry.kind is kind]

alias_map

alias_map(kind: TypeKind) -> dict[str, str]

{raw label: canonical label} for one kind — the operator-facing consolidation map.

Only genuine ALIASES appear (a type that was already canonical maps to itself and is omitted), so the map an operator reads on the Quality Report is exactly the set of rewrites the run performed — an empty map means nothing was consolidated.

Source code in packages/latence-core/src/latence_core/contracts.py

def alias_map(self, kind: TypeKind) -> dict[str, str]:
    """``{raw label: canonical label}`` for one kind — the operator-facing consolidation map.

    Only genuine ALIASES appear (a type that was already canonical maps to itself and is
    omitted), so the map an operator reads on the Quality Report is exactly the set of
    rewrites the run performed — an empty map means nothing was consolidated.
    """
    return {
        alias: entry.canonical
        for entry in self.types
        if entry.kind is kind
        for alias in entry.aliases
    }

canonical_for

canonical_for(kind: TypeKind, label: str) -> str

The canonical label for a raw label; the label itself when it is unknown.

Unknown-is-identity keeps the remap TOTAL: a label the vocabulary never saw (a delta run's brand-new type, a hand-written relation label) passes through unchanged rather than being dropped or mapped to a wrong cluster.

Source code in packages/latence-core/src/latence_core/contracts.py

def canonical_for(self, kind: TypeKind, label: str) -> str:
    """The canonical label for a raw ``label``; the label itself when it is unknown.

    Unknown-is-identity keeps the remap TOTAL: a label the vocabulary never saw (a delta
    run's brand-new type, a hand-written relation label) passes through unchanged rather
    than being dropped or mapped to a wrong cluster.
    """
    for entry in self.types:
        if entry.kind is kind and (label == entry.canonical or label in entry.aliases):
            return entry.canonical
    return label

merge

merge(other: TypeVocabulary) -> TypeVocabulary

Fold other into this vocabulary — append-don't-churn (the T4 delta contract).

For every cluster in other: if any of its labels already belongs to a cluster of the same kind here, that EXISTING cluster absorbs it — the existing canonical label is kept (renaming it would silently retype every node already in the KG), its aliases are unioned and strategies unioned. Otherwise the cluster is appended verbatim. When a cluster in other bridges two existing clusters, the first match (in this vocabulary's order) wins and the bridged labels join it — deterministic, and it never splits an existing cluster.

mention_count is the LATEST observation, not a running sum (T4). self is the vocabulary persisted by an earlier run and other the one a later run built over the whole CURRENT corpus, so the two count the same mentions: summing them would report a multiple of the true corpus salience on the Quality Report, and would grow without bound as deltas accumulate — the count would also differ between two applications of the same delta, breaking idempotency. So a cluster other observed takes other's count (several other clusters folding into ONE existing cluster do sum, because those are disjoint observations within one run), and a cluster other never observed keeps the count it was persisted with.

The result is ordered by (kind, canonical) and every alias list sorted, so merging is associative in outcome and byte-stable: the same pair of vocabularies always merges to the identical document.

Source code in packages/latence-core/src/latence_core/contracts.py

def merge(self, other: TypeVocabulary) -> TypeVocabulary:
    """Fold ``other`` into this vocabulary — append-don't-churn (the T4 delta contract).

    For every cluster in ``other``: if any of its labels already belongs to a cluster of the
    same kind here, that EXISTING cluster absorbs it — the existing canonical label is kept
    (renaming it would silently retype every node already in the KG), its aliases are unioned
    and strategies unioned. Otherwise the cluster is appended verbatim. When a cluster in
    ``other`` bridges two existing clusters, the first match (in this vocabulary's order) wins
    and the bridged labels join it — deterministic, and it never splits an existing cluster.

    **``mention_count`` is the LATEST observation, not a running sum** (T4). ``self`` is the
    vocabulary persisted by an earlier run and ``other`` the one a later run built over the
    whole CURRENT corpus, so the two count the same mentions: summing them would report a
    multiple of the true corpus salience on the Quality Report, and would grow without bound
    as deltas accumulate — the count would also differ between two applications of the same
    delta, breaking idempotency. So a cluster ``other`` observed takes ``other``'s count
    (several ``other`` clusters folding into ONE existing cluster do sum, because those are
    disjoint observations *within* one run), and a cluster ``other`` never observed keeps the
    count it was persisted with.

    The result is ordered by ``(kind, canonical)`` and every alias list sorted, so merging
    is associative in outcome and byte-stable: the same pair of vocabularies always merges
    to the identical document.
    """
    merged: list[dict[str, Any]] = [
        {
            "kind": e.kind,
            "canonical": e.canonical,
            "aliases": set(e.aliases),
            "mention_count": e.mention_count,
            "strategies": set(e.strategies),
        }
        for e in self.types
    ]
    # Which slots ``other`` has already contributed a count to in THIS merge — the first
    # contribution REPLACES the persisted count (same corpus, re-counted), later ones add to
    # it (two of ``other``'s clusters folding into one existing cluster are disjoint counts).
    recounted: set[int] = set()
    for entry in other.types:
        labels = set(entry.labels)
        target = next(
            (
                (index, slot)
                for index, slot in enumerate(merged)
                if slot["kind"] is entry.kind
                and (labels & ({str(slot["canonical"])} | set(slot["aliases"])))
            ),
            None,
        )
        if target is None:
            recounted.add(len(merged))
            merged.append(
                {
                    "kind": entry.kind,
                    "canonical": entry.canonical,
                    "aliases": set(entry.aliases),
                    "mention_count": entry.mention_count,
                    "strategies": set(entry.strategies),
                }
            )
            continue
        index, slot = target
        canonical = str(slot["canonical"])
        slot["aliases"] = (set(slot["aliases"]) | labels) - {canonical}
        slot["mention_count"] = (
            int(slot["mention_count"]) + entry.mention_count
            if index in recounted
            else entry.mention_count
        )
        recounted.add(index)
        slot["strategies"] = set(slot["strategies"]) | set(entry.strategies)
    return TypeVocabulary(
        types=[
            CanonicalType(
                kind=slot["kind"],
                canonical=str(slot["canonical"]),
                aliases=sorted(set(slot["aliases"])),
                mention_count=int(slot["mention_count"]),
                strategies=sorted(set(slot["strategies"])),
            )
            for slot in sorted(
                merged, key=lambda s: (str(TypeKind(s["kind"]).value), str(s["canonical"]))
            )
        ]
    )

RedactionAction

Bases: StrEnum

What a Redaction Provider did to a detected PII span (CONTEXT Redaction).

MASK replaces the span with a fixed mask token (e.g. [EMAIL]) — the default, lossy, irreversible. REPLACE substitutes a deterministic surrogate of the same type (a pseudonym) so the text stays readable while the real value is gone. HASH substitutes a stable salted hash so equal PII values remain joinable across the corpus without exposing the value. TAG leaves the surface text intact but records the span (detect-only) — used when policy for a Classification sensitivity is to flag PII for review rather than remove it.

PIISpan

Bases: BaseModel

One detected PII span in a document, with the action taken (CONTEXT Redaction).

Since W16 (ADR-0042) a PIISpan is carried on the :class:ChunkRecord the chunk-level Redactor produced: char_start/char_end are then offsets into that chunk's content (the coordinate system its masked_content is masked in — chunk-LOCAL, not document-global), while page_start/page_end still resolve to the ORIGINAL source page through the chunk's offset_map+page_slice. On the DEPRECATED document-level :class:RedactionRecord path the offsets are document-global (the historical shape below).

The structured, auditable record of a single redaction: what type of PII was found (pii_type, from the Provider's configured type set — the Capability names none), where it sat in the parent text (char_start/char_end — chunk-local content offsets on a ChunkRecord, or the document-offset coordinate system on the deprecated RedactionRecord — the coordinate every other Stage uses, so the span resolves back through the parent's page map to its source page), what was done (action), and the placeholder that replaced it in the masked variant (empty for a TAG action, which leaves the text intact). detector names the recogniser that fired (regex:email, gliner-pii) so a hybrid Provider's rule vs model provenance is auditable.

The raw PII value is deliberately NOT stored — a PIISpan carries the offset, the type, and the placeholder, never the sensitive surface string, so the audit trail itself never becomes a PII leak (S6 AC: no raw PII in logs/records).

RedactionRecord

Bases: Record

A per-document Redaction result — DEPRECATED, superseded by chunk-level redaction (W16).

.. deprecated:: W16 (ADR-0042) Document-level Redaction fed the WHOLE parsed document into a fixed PII model window (768 tokens for the gliner-family), so any PII past that point was SILENTLY truncated and leaked unmasked into the corpus. Redaction is now a chunk→chunk transform — the :class:~latence_core.capability.PIIDetector seam consumes :class:ChunkRecords and yields the same chunks with :attr:ChunkRecord.masked_content + :attr:ChunkRecord.pii_spans populated (each chunk ≤ the model window, so nothing is truncated). This RedactionRecord contract is RETAINED for the deprecated doc-level path (the provider redact_documents secondary method) and for prior-version JSON compatibility, but the blessed stacks and the Runner's redaction Stage now flow chunks.

A per-document Redaction result: the masked text variant + the PII spans (S6).

Redaction detects PII and produces masked/replaced variants of the text (CONTEXT Redaction). Like every inter-Stage carrier it is a :class:Record, so it carries Provenance and Classification and threads through the DAG / exports to the corpus — the masked_content is the PII-handled RAG-corpus text an Export materializes (ADR-0017). A RedactionRecord records:

• masked_content — the document's assembled markdown with every non-TAG PII span replaced by its placeholder (a TAG-only run leaves the text identical to the source but still records the spans).
• pii_spans — the structured, auditable list of every detected span (type, document offset, action, placeholder, page) — the S6 "structured record of detected PII spans" requirement. The raw PII value is never stored on a span.
• policy — the redaction policy name that was applied (keyed off the document's Classification.sensitivity, S6 AC), recorded so the run is auditable end to end.
• Provenance char_start/char_end — the covered document extent [0, len); the parent document's Classification, inherited unchanged.

document_record_id links the record back to the document it redacted; media_type mirrors the source (text/markdown). A document with no detected PII yields a RedactionRecord whose masked_content equals the source and whose pii_spans is empty (never an error) — so the corpus always has one PII-handled variant per document.

pii_count property

pii_count: int

Number of detected PII spans (0 for a clean document).

FeatureScope

Bases: StrEnum

The scope a :class:FeatureRecord describes (CONTEXT Profiling).

Profiling is a corpus-level Stage, so it emits at two scopes. DOCUMENT is one record per parsed document, carrying that document's per-document features (density, readability, Zipf α, compression ratio, structure). CORPUS is exactly one record per run, carrying the cross-document aggregate features (entity frequency, co-occurrence, type consensus, source coverage). Both are inter-Stage :class:Records — the split is a scope tag, not two contracts, so a consumer reads one exported stream and filters by scope.

DocumentFeatures

Bases: BaseModel

The fixed per-document statistical + quality feature set (S7).

Computed over one parsed :class:DocumentRecord's assembled markdown. This is the consolidated, fixed feature set the S7 issue calls for (the private stack's 9 enrichers RECAST into a lean closed set — no dynamic registry), so the schema is stable and a seeded run is byte-identical:

• char_count / word_count / sentence_count — raw size measures.
• unique_word_count — distinct lowercased word types (the vocabulary size).
• type_token_ratio — unique_word_count / word_count (lexical diversity; 0.0 for an empty document).
• density — non-whitespace characters as a fraction of all characters (0..1): how information-dense the text is versus padding/layout whitespace.
• readability — a normalised 0..1 readability score (higher = easier), derived from average sentence length and average word length (a Flesch-style reading-ease mapped into [0, 1]), dependency-free and deterministic.
• zipf_alpha — the fitted Zipfian exponent α of the word-frequency distribution (a well-formed natural-language corpus sits near 1.0); 0.0 when there are too few distinct words to fit.
• compression_ratio — compressed_size / raw_size under zlib (0..1-ish): a proxy for redundancy/repetitiveness (boilerplate compresses hard, so a low ratio flags a repetitive/templated document); 0.0 for empty text.
• avg_word_length / avg_sentence_length — the shape measures the readability score is built from, surfaced for downstream tuning.

EntityCoOccurrence

Bases: BaseModel

One cross-document co-occurrence: an ordered entity-text pair + its count (S7).

The pair is stored in a canonical order (left <= right lexicographically) so the same unordered pair is counted once regardless of the order the two mentions appeared in a document. document_count is the number of distinct documents in which both entities co-occur — the cross-document co-occurrence signal, not a raw within-document tally.

CorpusFeatures

Bases: BaseModel

The fixed cross-document statistical + quality feature set (S7).

Computed over the whole run — every parsed document and (when an Entity-Extraction Stage is upstream) every :class:EntityMention. The consolidated cross-document half of the RECAST enricher set:

• document_count — parsed documents profiled.
• total_words / total_unique_words — corpus size + vocabulary.
• corpus_zipf_alpha — the Zipfian exponent fitted over the whole-corpus word-frequency distribution (the corpus-level Zipf α the issue names).
• language_coverage — per-Classification.language document counts (source coverage by language).
• category_coverage / sensitivity_coverage — the same source-coverage breakdown by content category and sensitivity.
• entity_frequency — cross-document entity mention counts, keyed by the mention surface text (how often each entity appears corpus-wide).
• entity_type_frequency — mention counts keyed by entity label.
• entity_type_consensus — per entity text, the fraction of its mentions that carry its single most-common label (1.0 = every mention agrees on the type; a lower value flags an entity the extractor typed inconsistently across the corpus). Only entities seen at least twice are scored.
• co_occurrences — the top cross-document entity co-occurrence pairs (:class:EntityCoOccurrence), most-frequent first, capped for a bounded report.

Every map is emitted in a deterministic order (sorted), so a seeded run is byte-identical.

FeatureRecord

Bases: Record

A corpus-level Profiling result: a typed FeatureSet at one scope (CONTEXT Profiling).

Profiling computes statistical and quality features over the corpus. A FeatureRecord is an inter-Stage :class:Record — it carries Provenance and Classification like every other carrier — so it threads through the DAG, is checkpointed/resumed uniformly, and exports directly. Exactly one of document/corpus is populated, selected by scope:

• scope == DOCUMENT: document holds the per-document :class:DocumentFeatures and document_record_id links back to the profiled DocumentRecord; its Provenance is that document's lineage, its Classification inherited unchanged. corpus is None.
• scope == CORPUS: corpus holds the cross-document :class:CorpusFeatures; document_record_id is None and its Provenance is the synthetic corpus-scope lineage (the whole run is its source). document is None.

The scope/payload consistency is contract-enforced, so a Provider cannot emit a document-scope record with corpus features (or vice-versa) or leave both empty.

Evidence

Bases: BaseModel

The justification linking an inference back to its source mentions (CONTEXT Evidence).

Evidence is the record linking a knowledge-graph edge — or, in S8, a Disambiguation merge/link decision — back to the specific source mentions and offsets that justify it, with a confidence value. It is deliberately distinct from :class:Provenance: Provenance is source lineage (which file, which page, which char span a record came from); Evidence is justification for an inference (which mentions made us believe two surface strings name the same entity, and how strongly). A merge that folds two mentions into one canonical entity carries the Evidence for why — so no merge is a silent, unexplained act.

mention_ids are the record_ids of the source :class:EntityMentions (or, for a relation, the endpoints) the inference rests on; document_ids the distinct source documents they came from (the cross-document reach of the evidence); snippet an optional short human-readable justification (never raw sensitive content — a surface string or the strategy name), and confidence the 0..1 strength of the justification.

MergeStrategy

Bases: StrEnum

Which resolver strategy justified a merge/link decision (ported EntityResolver cascade).

The ported resolution/entity_resolver.py asset — taken as-is — resolves entity mentions with a 5-strategy cascade, tried in order of descending precision so the strongest available signal wins and the reason is auditable:

• EXACT — identical normalised surface strings (case/whitespace-folded).
• ALIAS — a configured alias/synonym maps one surface to another's canonical.
• ACRONYM — an acronym matches the initials of a multi-word name (IBM ↔ International Business Machines).
• SUBSTRING — one name is a whole-word substring of the other (Acme ↔ Acme Corporation), gated by a length ratio so a ⊄ about.
• EMBEDDING — a vector-similarity match above a threshold (the CPU reference Provider has no embedder, so this strategy is inert there and is the pluggable FAISS/GPU upgrade path, S8 AC).

KB_LINK is the linking (not merge) disposition: the GLinker pipeline matched a mention to a canonical entity in an external KB (L2→L3→L4→L0). Recorded as the justification on a :class:CanonicalEntity that carries a kb_id.

MergeDecision

Bases: BaseModel

One audited, confidence-weighted merge/link decision (no silent over-merge, S8 AC).

The ported EntityResolver keeps an audited merge log: every candidate merge — applied or not — is recorded with the strategy that proposed it, its confidence, and the Evidence, so a run never silently over-merges. A decision below the confidence policy is recorded with applied == False and is NOT folded into a canonical entity — the S8 acceptance criterion "merges below the confidence policy are not applied and are logged".

source_text/target_text are the two surface strings the decision related (source folded into target); strategy the cascade rung that proposed it; confidence its 0..1 score; applied whether the confidence policy accepted it; evidence the justification (the mentions/offsets). reason is a short human-readable note (e.g. "acronym: IBM ~ International Business Machines").

CanonicalEntity

Bases: Record

A canonical entity: a merged cross-document cluster of mentions (CONTEXT Disambiguation).

Disambiguation links entity mentions to canonical entities and merges duplicates across documents. A CanonicalEntity is the entity-scope output: one per resolved cluster, carrying the chosen canonical surface (canonical_text) and entity_type, the member_mention_ids (the record_ids of every :class:EntityMention folded into it), the source_document_ids it spans (its cross-document reach — the affected-set key S11 keys incremental recompute off), the merge_decisions that built it (the audit trail), and its :class:Evidence. When the cluster linked to an external KB, the matched id is recorded in kb_id (None for an unlinked entity — the graceful fallback, S8 AC).

Like every inter-Stage carrier it is a :class:Record, so it carries Provenance and Classification and threads through the DAG / exports. Its Provenance is the canonical mention's lineage — page-accurate via :class:~latence_core.pagemap.PageOffsetIndex, so a downstream KG edge can cite "page 7 of the contract", the S8 page-map-integration sub-task — and its Classification is inherited from that mention. mention_count is the size of the cluster; a singleton entity (no duplicate found) is valid — it is a canonical entity of one mention, not an error.

mention_count property

mention_count: int

The number of source mentions folded into this canonical entity.

linked property

linked: bool

True if this entity linked to an external KB (carries a kb_id).

NormalizedRelation

Bases: Record

A relation remapped onto canonical entities, label normalised (ported RelationNormalizer).

The ported relations/normalizer.py asset, taken as-is: GLinker-style relation L2 resolution — fuzzy + alias label normalisation, type filtering, and inverse detection + direction swap (an employed_by relation whose configured inverse is employs is rewritten to the canonical direction). A NormalizedRelation is the relation-scope Disambiguation output: it takes a source :class:RelationMention and rewrites its endpoints from mention ids to the head_entity_id/tail_entity_id of the :class:CanonicalEntitys they resolved to, and its label to the normalised canonical relation label. inverted records whether the head/tail were swapped to reach the canonical direction (so the rewrite is auditable). source_relation_id links back to the original RelationMention.

Like every carrier it is a :class:Record (Provenance + Classification, inherited from the source relation), so it threads through the DAG and exports. A relation whose head or tail did not resolve to a canonical entity is dropped by the Provider (not emitted as a broken relation) and counted in the Quality Report — never silently mis-linked.

DisambiguationScope

Bases: StrEnum

The scope a :class:DisambiguationRecord describes (CONTEXT Disambiguation).

Disambiguation is a corpus-level Stage, so — like Profiling (ADR-0025) — it emits a single scoped carrier rather than two contracts. ENTITY records carry one :class:CanonicalEntity each (the entity-linking + merge half); RELATION records carry one :class:NormalizedRelation each (the relation-normalisation half). A consumer reads one exported stream and filters by scope; the DAG, the checkpoint, and Export all treat one carrier.

DisambiguationRecord

Bases: Record

A corpus-level Disambiguation result: one canonical entity OR one normalized relation.

The scoped inter-Stage carrier for the Disambiguation Stage, mirroring :class:FeatureRecord's scope split (ADR-0025/0026). Exactly one of entity/relation is populated, selected by scope:

• scope == ENTITY: entity holds the :class:CanonicalEntity; relation is None. Its Provenance is the canonical mention's page-accurate lineage and its Classification is inherited from that mention.
• scope == RELATION: relation holds the :class:NormalizedRelation; entity is None. Its Provenance/Classification are the source relation's.

The scope/payload consistency is contract-enforced, so a Provider cannot emit an entity-scope record with a relation payload (or leave both empty). Records are emitted deterministically (all ENTITY records in canonical-text order, then all RELATION records) so a seeded run is byte-identical (Baseline bar).

GraphNode

Bases: Record

One node in the canonical knowledge graph (CONTEXT Graph Assembly).

Graph Assembly builds the canonical knowledge graph — nodes, edges, and per-edge Evidence — from disambiguated entities and relations. A GraphNode is the graph's entity half: exactly one node per :class:CanonicalEntity, carrying a deterministic node_id (content-addressed: a sha256 over the corpus id and the canonical entity id, so the same entity always yields the same node id and two runs over the same corpus produce the same graph — the S9 "deterministic IDs" acceptance criterion), the canonical_name and label (entity type), the entity_id of the source :class:CanonicalEntity, the member_mention_ids folded into it, its source_document_ids (cross-document reach), an optional external kb_id link, and the :class:Evidence back to the source mentions that justify it.

Like every inter-Stage carrier it is a :class:Record, so it carries Provenance (inherited from the canonical entity — page-accurate via PageOffsetIndex) and Classification. properties is an open bag of node attributes materialised into the KG export (confidence, mention_count, document_count), kept JSON-serialisable so it round- trips through Parquet/TTL/GraphML.

linked property

linked: bool

True if this node links to an external KB (carries a kb_id).

GraphEdge

Bases: Record

One edge in the canonical knowledge graph, with per-edge Evidence (S9 Graph Assembly).

A GraphEdge is the graph's relation half: exactly one edge per :class:NormalizedRelation, carrying a deterministic edge_id (content-addressed: a sha256 over the corpus id and the source relation id), the source_node_id / target_node_id it connects (the deterministic ids of the head/tail :class:GraphNodes), the normalized relation label, its confidence, and its :class:Evidence — the record linking the edge back to the specific source mentions and offsets that justify it, with a confidence value (CONTEXT Evidence, reused verbatim from S8 for per-edge justification: every edge carries Evidence, the S9 acceptance criterion). relation_id back-references the source :class:NormalizedRelation.

Like every inter-Stage carrier it is a :class:Record (Provenance + Classification, inherited from the source relation). The head and tail node must be distinct (no self-loop). properties is an open bag of edge attributes materialised into the KG export (inverted, source_document_ids), kept JSON-serialisable.

HyperedgeRelation

Bases: BaseModel

One directed relation carried inside a :class:GraphHyperedge (CONTEXT Hyperedge).

The Hyperedge's relation signature is the set of these triples — (head_node_id, label, tail_node_id) — and two Hyperedges are dedup-equivalent only when BOTH their entity sets AND their relation signatures match (ADR-0057: identical entity sets with differing signatures are distinct facts about the same group and are never merged). relation_id back-references the source relation record for audit; confidence is the extractor's score, one input of the Hyperedge's gamma.

signature property

signature: tuple[str, str, str]

The dedup signature triple: (head_node_id, label, tail_node_id).

HyperedgeSourceRef

Bases: BaseModel

A non-primary source span folded into a Hyperedge on dedup-merge (CONTEXT Hyperedge).

On merge the highest-γ instance's tau_text and Provenance stay primary and every other instance is appended here (ADR-0057 / spec 1.4), so a corpus-repeated fact keeps its full audit trail while support counts the repetitions. Offsets are in the parent document's original coordinates, like every Provenance span.

GraphHyperedge

Bases: Record

One Hyperedge: the m-ary unit of retrieval (CONTEXT Hyperedge, ADR-0057).

Where a :class:GraphEdge connects exactly two nodes, a Hyperedge connects the FULL set of canonical entities co-occurring in one minimal text span, together with the directed relations extracted from that span — so a joint fact stays one selectable unit instead of fragmenting into disconnected pairs. It is the atomic item the chain-of-evidence selector packs into a token budget, which is why its selection inputs are precomputed on the record (ADR-0056: the index is built for the solver):

• tau_text — the verbatim span text, in the corpus redaction variant (ADR-0057): it never contains text the exported corpus would not contain. The span's Provenance stays in original coordinates for audit and Purge.
• token_cost — the knapsack weight, computed at build time by the named token_counter and treated downstream as an approximate ranking weight whose budget guarantee comes from an exact query-time verification pass (ADR-0058). The counter identity is stored beside the value so a later mismatch is detectable, never silent.
• gamma — extraction confidence composed from link/relation/mention confidences (spec 1.5; uncalibrated until the calibration decision lands).
• support — how many distinct source spans asserted this same fact (dedup-merge counter); a frequency prior for scoring.
• has_relations — False marks a weak Hyperedge (entities only), emitted behind a config toggle and utility-discounted downstream.

hyperedge_id is deterministic and content-addressed over the corpus id, the sorted entity-node set, and the sorted relation signature — NOT over the span position — so the same fact found twice hashes to the same id and dedup-merge is the identity operation the S11 affected-set machinery can key off. Like every carrier it is a :class:Record (Provenance = the primary span, original coordinates; Classification inherited), and it carries :class:Evidence linking back to the member mentions.

arity property

arity: int

The number of entities this Hyperedge connects (m).

weak property

weak: bool

True for an entities-only Hyperedge (no extracted relation).

GraphScope

Bases: StrEnum

The scope a :class:GraphRecord describes (CONTEXT Graph Assembly).

Graph Assembly is a corpus-level Stage, so — like Disambiguation (ADR-0026) and Profiling (ADR-0025) — it emits a single scoped carrier rather than two contracts. NODE records carry one :class:GraphNode each (the graph's entity half); EDGE records carry one :class:GraphEdge each (the graph's relation half); HYPEREDGE records carry one :class:GraphHyperedge each (the m-ary retrieval unit, ADR-0057 — emitted only when the Stage is configured for hyperedges, so a stack that never opts in never sees the scope). A consumer reads one exported stream and filters by scope; the DAG, the checkpoint, and the KG Export all treat one carrier.

GraphRecord

Bases: Record

A corpus-level Graph-Assembly result: one graph node OR one graph edge.

The scoped inter-Stage carrier for the Graph Assembly Stage, mirroring :class:DisambiguationRecord's scope split (ADR-0026). Exactly one of node/edge is populated, selected by scope:

• scope == NODE: node holds the :class:GraphNode; edge/hyperedge are None. Its Provenance/Classification are the source canonical entity's (page-accurate).
• scope == EDGE: edge holds the :class:GraphEdge; node/hyperedge are None. Its Provenance/Classification are the source normalized relation's.
• scope == HYPEREDGE: hyperedge holds the :class:GraphHyperedge (ADR-0057); node/edge are None. Its Provenance is the primary span's, in original coordinates.

The scope/payload consistency is contract-enforced, so a Provider cannot emit a node-scope record with an edge payload (or leave both empty). Records are emitted deterministically (all NODE records in node-id order, then all EDGE records in edge-id order, then any HYPEREDGE records in hyperedge-id order), so a seeded run is byte-identical (Baseline bar).

DeltaKind

Bases: StrEnum

How a document changed since the last committed Corpus Version (CONTEXT Delta).

A :class:Delta is a content-hash diff of the Source manifest against the last committed Corpus Version (ports delta_detector). Each document falls into exactly one class:

• ADDED — a document id not present in the parent Version (new source).
• UPDATED — the same document id, a new content hash (the text or its derived extraction changed): doc-level Stages re-run on it, its old derived records are retracted.
• DELETED — a document id in the parent Version absent from the new manifest: its records are retracted (or purged), affected clusters re-resolved.
• UNCHANGED — same id, same hash: nothing re-runs; its committed records are carried forward verbatim.

When is UPDATED reachable? (issue #42). UPDATED fires only when a document id persists across two manifests with a changed content hash — i.e. the Source assigns a stable identity independent of content. The reference source.local_folder (:class:~latence_core.stages.source.LocalFolderSource) is content-addressed: a document's id IS sha256 of its bytes, so any in-place edit mints a new id and retires the old one, and the detector classifies that as ADDED (new id) + DELETED (old id) — never UPDATED. So UPDATED is unreachable under the content-addressed local-folder Source and is exercised only by a Source with content-independent stable ids (e.g. a path-keyed or database-keyed Source). It is retained (not removed) because it is a valid, contract-versioned classification the DeltaProcessor and DeltaRunner._commit already handle, and a stable-id Source is a supported Source variant; removing it would be a lossy contract change.

DeletionMode

Bases: StrEnum

The disposition for a deleted document (CONTEXT Retraction / Purge).

Deletion is dual-mode (ADR-0018). RETRACTION is the default soft tombstone: the document and its derived records are excluded from every Export, the knowledge graph, and the RAG corpus, but retained for audit and rollback, with affected clusters re-resolved. PURGE is the hard GDPR / right-to-be-forgotten erasure: physical removal of the source, all derived records, and their Evidence, followed by reconciliation of the affected set — nothing derived survives.

DeltaEntry

Bases: BaseModel

One document's classification within a :class:Delta (CONTEXT Delta).

The content-hash diff assigns each document a :class:DeltaKind. content_hash is the deterministic SHA-256 the ported detector computed over the document's markdown plus its derived entity texts and relation labels (None for a DELETED document, which no longer has a current hash). file_name is carried for the audit; identity is the document_id alone (never the file name).

Delta

Bases: BaseModel

The classified change set since the last committed Corpus Version (CONTEXT Delta).

Computed as a content-hash diff of the Source manifest against the parent Version's manifest (ports delta_detector): every document is an ADDED / UPDATED / DELETED / UNCHANGED :class:DeltaEntry. parent_version is the Corpus Version this delta was computed against (None for the very first, bootstrap delta against an empty corpus). deletion_mode records how deleted documents are being handled (Retraction by default, Purge on request). intentional_deletion records whether this delta's DELETED entries are user-requested tombstones (an explicit :meth:retract/:meth:purge) rather than the incidental delete-half of an ordinary in-place content edit: because document ids are content-addressed, editing a file mints a new id and retires the old one, so the detector classifies the old content-id DELETED even though the user retracted nothing. Only an intentional_deletion DELETED id is a durable tombstone that stays excluded at Source on every future run; an edit's incidental DELETED id must NOT be permanently blacklisted, or an edit-then-revert to earlier content would be silently dropped. drift is the cheap drift metric (fraction of the corpus changed), and triggered_reconciliation records whether it crossed the threshold and forced a full Reconciliation — so the trigger and its outcome are auditable (the S11 AC).

changed property

changed: list[str]

The documents a delta must re-process (added + updated), sorted.

is_empty property

is_empty: bool

True if nothing changed (no add/update/delete) — an idempotent no-op re-run.

CorpusVersion

Bases: BaseModel

An immutable, numbered committed state of the corpus (CONTEXT Corpus Version).

A Delta run reads Version N and transactionally produces Version N+1 (via the ported WAL + atomic manifest swap); any prior Version is inspectable for audit and rollback. A CorpusVersion is the manifest of that committed state: its version number, the parent_version it succeeded (None for the initial Version 0), the :class:Delta that produced it, the live record_count after the delta, a deterministic fingerprint (a content hash over the committed record ids — identical input + config ⇒ identical fingerprint, the G1 determinism criterion), and reconciled (True when this Version was produced by a full Reconciliation rather than an incremental affected-set recompute). created_ts is excluded from the fingerprint so re-running a delta is byte-identical apart from the timestamp.

DeltaChurn

Bases: BaseModel

The delta run's churn roll-up for the Quality Report (S11 AC).

"Delta churn (docs ±, entities created/merged/split, edges ±, drift) lands in the Quality Report" — the S11 acceptance criterion, kept to counts (no raw text), so it is safe to persist. documents_added / documents_updated / documents_deleted classify the delta; documents_unchanged are carried forward untouched. entities_created / entities_merged / entities_split are the affected-set recompute's effect on canonical entities (a bridging doc that MERGES two clusters, a deleted doc that SPLITS one); edges_added / edges_retracted the graph patch. records_retracted / records_purged count the soft-tombstoned vs physically-erased derived records. canonical_types_added / type_aliases_added / canonical_types_total are the T4 type-vocabulary churn: how the delta EXTENDED the persisted canonical type vocabulary (new canonical entries vs raw labels that aliased into an existing one). drift is the metric and reconciliation_triggered whether it forced a full recompute, with deletion_mode recording how deletes were handled. from_version / to_version bracket the transaction.

affected_set_size / corpus_record_count are the perf-baseline witness that the recompute was affected-set-scoped, not O(corpus): on an incremental (non-reconciliation) delta affected_set_size (the corpus-level records the blocking-neighborhood recompute actually re-resolved) is <= corpus_record_count (the whole current corpus-level set), and a genuinely local change touches only a small fraction — the measurable ADR-0018 "recompute only the affected set" bound. On a full Reconciliation the two are equal.