Skip to content

The Quality Report is promoted to the full first-class artifact; the gold-set harness is a Runner-side evaluation keyed by file name, not a DAG Stage

S10 promotes the Quality Report (CONTEXT) from the skeleton that grew slice-by-slice (S1–S9) to the full first-class artifact ADR-0008 called for: contract-completeness now covers offsets aligned (not just Provenance/Classification present), a corpus-level drift roll-up surfaces Provenance-integrity across every derived offset, the report renders a human-readable summary alongside its machine-readable JSON, and — the headline — a gold-set harness reports per-Stage precision/recall/F1 on the adopter's OWN corpus. Two design questions had non-obvious answers.

The gold-set harness is a Runner-side evaluation over the report inputs, not a Stage/Capability

Every prior slice added a Stage with a Capability protocol, a reference Provider, and a scoped inter-Stage Record that flows through the DAG (S4 EntityMention, …, S9 GraphRecord). The gold-set harness is deliberately not one of those. It does not consume or produce a carrier the DAG threads; it does not run as a node in topological order; there is no Scorer Capability and no latence.providers entry point for it. It is a pure, deterministic evaluation the Runner performs while building the report — it reads the same extracted mentions/relations the report already aggregates, matches them against the adopter's annotations, and writes the scores into the Quality Report.

Why. A gold set is reference data about the run, not data the pipeline transforms. Modelling it as a Stage would force artificial questions — what does the "scoring Stage" emit downstream? where in the DAG does it sit relative to Export? — that have no honest answer, because scoring is orthogonal to the pipeline's data flow. ADR-0008 already frames the harness as part of the Quality Report artifact ("an optional gold-set harness lets an adopter drop in 20–50 annotated documents and get precision/recall per Stage"), not as a new pipeline capability. Keeping it Runner-side also means it recomputes identically on a resumed run (from the reloaded checkpoint records) with no extra checkpoint of its own, and a pipeline with no gold set pays exactly nothing. The scoring itself lives in a pure latence_core.goldset engine module (like graph.py / disambiguation.py), so it is unit- testable in isolation and reusable, while the Runner owns only the wiring.

Rejected: a Scorer Capability + Provider (invents a Stage with no carrier and no DAG position); writing scores to a separate side-car file (splits the auditable "AI-ready" substantiation away from the report that ADR-0008 says is that substantiation).

Annotations bind by source file name; matching is offset-aware, label-sensitive, one-to-one

The gold set is a versioned Pydantic contract (GoldSetGoldDocumentGoldEntity / GoldRelation), authored as JSON or YAML (one-schema/dual-serialization, ADR-0006) and loaded from Storage via an optional gold_set_uri on the Pipeline (PIPELINE_SCHEMA_VERSION → 2; additive, backward-compatible). Each GoldDocument is keyed by file_name, not the content-addressed document_id the run computes — the file name is the stable key the adopter knows when authoring annotations, and the harness resolves the run's records back to it through their Provenance. Annotations a document the Source never read simply go unscored, and the gap (documents_matched < documents_annotated) is visible in the report.

Matching is deterministic (Baseline bar): a prediction matches a gold annotation when their labels are equal AND their character spans overlap by at least a configurable Jaccard threshold (match_min_overlap; 0.0 = any overlap). Entity assignment is a maximum-cardinality one-to-one matching (augmenting paths / Kuhn's algorithm, with deterministic tie-breaking) — never a single-pass greedy, which would under-count precision and recall by letting one prediction steal a gold span a later prediction was its only match for (#76). Relation assignment is likewise one-to-one and maximum-cardinality over the pairs whose both endpoints resolve. Each gold annotation and each prediction is consumed at most once, so precision and recall are well-defined and never double-count, and the matcher never leaves a matchable pair unmatched. A relation matches only when its label matches AND both endpoints resolve (by span + label) to the gold relation's head and tail entities — a label-only match is not credited. Scores are computed overall and per label; every derivation guards its empty denominator (no predictions ⇒ precision 0.0; no gold ⇒ recall 0.0), so the report never divides by zero. The result is byte-identical across seeded runs.

Completeness gains offset alignment; drift is a cross-Stage roll-up

CompletenessCheck now audits that every offset-bearing record's char span is well-ordered and, when it carries page coordinates, that those are well-ordered too — the "offsets aligned" half of the S10 contract-completeness AC. Page presence is not re-required here: the sub-document carriers whose contracts already require a page span (EntityMention / RelationMention) guarantee it at construction, while a whole-document carrier legitimately holds a [0, len) char span with no page span. Separately, a corpus-level DriftDiagnostics rolls up the ported PageOffsetIndex drift across every page-map-bearing chunk (a finer, cross-Stage view than the Parse-only page-boundary audit): clean means every derived offset resolves exactly to its source page. Both are counts only (no raw text), safe to persist.