Skip to content

Incremental corpus deltas: affected-set recompute, drift-triggered reconcile, dual-mode deletion

The corpus is not rebuilt on every change. A delta is computed as a content-hash diff of the Source manifest against the last committed Corpus Version — classifying each file as added, updated (same identity, new hash), deleted, or unchanged (ports delta_detector). Processing then splits by Stage altitude:

  • Doc-level Stages (Parse … Redaction) are purely incremental: only added/updated documents run; records for deleted/superseded documents are retracted. Doc-level records are keyed by document identity and independent, so this is exact.
  • Corpus-level Stages (Disambiguation, Graph Assembly) recompute only the affected set — the blocking neighborhoods touched by changed mentions plus their immediate neighbors. This catches the cases the private stack's "link new mentions against the existing KB" missed: two pre-existing entity clusters that should MERGE when a bridging document arrives, and a cluster that should SPLIT when its only bridging document is deleted. Retraction is surgical via Evidence provenance (source_document_ids), porting graph/patcher.
  • Because incremental resolution is order-dependent, a cheap drift metric (fraction of the corpus changed since the last full pass, or affected-block fanout exceeding a threshold) triggers a full reconciliation that recomputes all blocks. This is the lean/enterprise hinge: incremental speed with a bounded-error backstop.

Deletion is dual-mode. The default DELETE is a soft Retraction: the document and its derived records are tombstoned — excluded from every Export, the knowledge graph, and the RAG corpus, retained for audit and rollback — and affected clusters are re-resolved. A separate Purge (GDPR / right-to-be-forgotten) physically removes the source, all derived records, and their Evidence, then reconciles the affected set. Evidence provenance makes both operations surgical rather than full-corpus scans.

Erasure re-writes the manifest (H-D1, see ADR-0029 §5/§5a). Corpus Versions are otherwise immutable audit history, but a Purge is the one operation that trumps that immutability: the "nothing derived survives" criterion requires expunging a purged document's records from every records.jsonl under the corpus root — committed and the abandoned WAL staging dirs a crash-mid-commit leaves behind — and re-fingerprinting/re-counting each rewritten Version's version.json so the head's stored fingerprint (which drives the idempotency check) reflects the erased reality, not the pre-erasure one. Immutability is reconciled with erasure by scope: a Retraction retains and never rewrites; a Purge rewrites the manifest to the erased truth (immutable delta/parent/timestamp lineage preserved).

Every delta run is a transaction. It reads Corpus Version N and produces an immutable Version N+1 via the write-ahead log and an atomic manifest swap (ports the WAL): re-running the same delta is idempotent, a failed run leaves N intact, and any prior Version is inspectable for audit and rollback.

Rejected: full corpus-level recompute every delta (correct but O(corpus) per change — fine at 10k–100k docs, untenable at millions), pure incremental with no reconciliation (resolution quality silently drifts — indefensible for a "high-quality data" claim), append-only with no merge/split (KG degrades as same entities stay split), soft-retract-only (cannot satisfy GDPR/CCPA erasure — a blocker for the regulated buyers this targets), hard-purge-only (no rollback, no audit history, accidental deletes unrecoverable), and deferring deletion past v1 ("we can't remove data" is an instant enterprise disqualifier). The RotatE incremental warm-start trainer from the private stack stays out of v1 with the rest of link prediction (ADR-0005).

Delivered scope vs. the efficiency promise (issue #42, honest boundary)

The affected-set property described above holds for the committed corpus-level record set, and only there — this is the part DeltaProcessor owns. Given the parent Version's records + the current run's freshly-extracted records, the reference delta.affected_set Provider re-resolves only the blocking-neighborhood transitive closure (compute_affected_set, state/incremental.py) the delta touched, carries every untouched cluster forward byte-identical, and records the affected-set size vs the whole corpus (DeltaChurn.affected_set_size / corpus_record_count) as the measurable perf witness. So the stored corpus, and what a delta re-writes, is genuinely delta-scoped.

What is not yet incremental is the upstream extraction pass. Each delta run uses a fresh run_id, so DeltaRunner re-parses / re-chunks / re-extracts / re-disambiguates the whole current source through the Runner before the DeltaProcessor scopes the committed set — there is no cross-run doc-level checkpoint reuse. The doc-level extraction cost of a delta is therefore still O(corpus), not O(delta). Making doc-level Stages skip unchanged documents across runs (true incremental extraction) is a Runner-core change with blast radius on offsets / page-provenance / Evidence / idempotency; it is tracked as post-G1 follow-up work and is deliberately not delivered in the S11 slice. The docstrings on DeltaRunner, AffectedSetDeltaProcessor, and this ADR state this boundary explicitly rather than implying the whole delta path is O(delta).