Skip to content

A second CPU reference Provider for each single-provider Capability

The Hardening bar (DEFINITION-OF-DONE → "Providers: ≥2 reference providers per Capability", issue #77) requires every Capability to ship with more than one reference Provider behind its seam, with parity tests (same typed contracts) and a benchmark vs the existing one — the concrete proof that the Capability really is a model-agnostic seam and not a single implementation wearing a Protocol. Five Capabilities shipped with exactly one in-core Provider (Chunk, Content Screening, Profiling, Disambiguation, Graph Assembly); the Entity/Relation/Redaction/Parser/Embedder Capabilities already had a learned or endpoint second Provider in a sibling package (ADR-0012/0013/0024/0030), and Source/Export are thin IO seams. This ADR records the second CPU Provider chosen for each of the five, and why each is a genuinely different strategy rather than a renamed clone.

Each second Provider is CPU-first, zero-dep, deterministic, and behind the existing seam (ADR-0004/0007/0016). They ship in latence-core alongside the reference Providers, register under the same latence.providers entry-point group (chunk.sentence_window, screening.content_fuzzy, profiling.lightweight, disambiguation.exact_surface, graph.weighted), and satisfy the identical typed Protocol — so a consumer swaps to one by name with no pipeline, contract, or code change (the one-line Provider swap, ADR-0011). None introduces a model, so none needs a weights/code license note (ADR-0012); the heavy/learned alternatives remain the sibling-package upgrade path. Every correctness property the reference Provider guarantees is preserved and re-asserted on the second Provider: offset + page-provenance round-trip (Chunk), the fixed streaming feature set (Profiling), audited no-silent-over-merge + page-accurate Provenance + Evidence (Disambiguation), deterministic content-addressed ids + per-edge Evidence + endpoint validation (Graph Assembly), and marker-propagation + flag-not-drop (Screening).

The five strategies, each a real alternative measured against the reference. - Chunk — SentenceWindowChunker: greedy whole-sentence packing with no token overlap, vs the reference's token-window + best-break with overlap. It never ends a chunk mid-sentence (unless one sentence exceeds the budget), trading cross-boundary retrieval overlap for sentence-coherent passages. It reuses the same strip_markup offset map (a new engine entry point chunk_document_by_sentence), so the S3 chunk → offset → original-page round-trip holds identically — the benchmark asserts the sentence-aligned boundaries as a hard bound. - Content Screening — FuzzyInjectionContentScreener: an obfuscation-tolerant injection detector that compacts each chunk to a canonical alphanumeric-only form and matches compacted signatures, catching separator-padded evasions (i g n o r e p r e v i o u s, ig.no.re/pre.vi.ous) the reference keyword regexes miss. Because Screening is a Gate-bar security Stage, the compaction is a single linear pass with no regex and no backtracking — ReDoS-proof. The DoS bound caps the number of alphanumeric characters kept (_FUZZY_SCAN_CAP, a bound on the compacted output), not a raw prefix: an early raw-prefix cap was defeated by leading separator-padding — cost-free whitespace/punctuation is free under the token budget, so a chunk of 8192 spaces + an injection survives chunking as one chunk, yet a raw-prefix slice removed only the padding and missed the injection (#77 security review). Capping the compacted length means free separators do not consume the budget, closing that evasion; a secondary _FUZZY_RAW_SCAN_CAP bounds the raw characters iterated so a pathological all-punctuation megabyte chunk (which never fills the alphanumeric budget) is still O(1)-bounded (asserted adversarially). A regression test (test_second_screener_not_evaded_by_leading_separator_padding) locks the fix. It flags-not-drops by default, mirroring the reference. - Profiling — LightweightProfiler: a scale-oriented profiler that keeps the reference's per-document feature set and the cheap O(mentions) corpus tallies (entity/type frequency, type consensus) but skips the O(distinct-entities²) co-occurrence pairing entirely, so its corpus record's co_occurrences is always empty — a strictly-linear-in-mentions pass for a huge, hostile-fan-out corpus. Parity is asserted on the shared features; the empty co-occurrence is the measured difference. - Disambiguation — ExactSurfaceDisambiguator: the classic exact-blocking baseline a fuzzy resolver is measured against. It restricts the resolver to the two highest-precision rungs — exact normalized-surface equality and configured aliases — via a new exact_only flag on the ported EntityResolver, disabling the acronym/substring/embedding rungs (and their below-policy audits). Higher precision, lower recall: it will not unify "IBM" with "International Business Machines" without an explicit alias, and can never fuzzy-over-merge. Relation normalization + KB linking are shared with the reference (relation/linking parity). - Graph Assembly — WeightedGraphAssembler: folds every parallel same-(head, tail, label) relation into one weighted edge (a simple weighted graph), vs the reference's one-edge-per-relation multigraph. The folded edge's confidence is the max over the group, its properties["weight"] the multiplicity, and its per-edge Evidence the union of the folded relations' mention + document ids (so it still cites every justifying source). Its edge_id is content-addressed over the endpoint pair + label (deterministic across runs); nodes are assembled by the same ported GraphBuilder (node parity).

Rejected: renamed clones (fail the seam's purpose — a second name for one implementation proves nothing), a config-preset masquerading as a Provider (the difference must be a real strategy a benchmark can distinguish), and pulling a learned/GPU alternative into core for the count (breaks CPU-first/thin-core; the learned second Providers already live in sibling packages). Two small, safe engine additions were made to keep the second Providers thin rather than duplicating logic: chunk_document_by_sentence (reuses the strip offset map) and the EntityResolver(exact_only=...) flag (short-circuits after the alias rung, emitting no spurious fuzzy audit).