Skip to content

Parse carries a page map and an error disposition on the record contract

S2 makes Parse a real Stage: a document Parser that turns PDFs (and text) into markdown and must (a) preserve which source page any character came from, and (b) survive a corrupt or type-spoofed file without aborting the run. Two design questions had non-obvious answers, recorded here.

PageMap is a latence-core contract; PageOffsetIndex is a core resolver — not a Provider concern

The ported PageOffsetIndex asset — the predecessor implementation's provenance-integrity piece, landing here in S2/S8 — is offset→page resolution: it is used at Parse to build the map, and again downstream at Chunk (offset-preserving splits), Entity Extraction (mention spans), and Disambiguation (Evidence offsets → source page). If the PageMap contract and the resolver lived in the latence-parser-document package, every downstream Stage would depend on a Provider package to interpret Provenance — inverting the dependency arrow (core must not depend on a plugin) and coupling the corpus Stages to one Parser's package.

Decision. PageSpan / PageMap are versioned contracts in latence_core.contracts (bumping SCHEMA_VERSION to 2), and PageOffsetIndex + PageDriftDiagnostics live in latence_core.pagemap. A DocumentRecord gains an optional page_map. The Parser Provider produces the map (via PageMap.from_page_texts, which assembles the markdown and the boundaries together so they are exact by construction); core owns the type and the resolver, so any Stage resolves an offset to a page without importing a Provider. The Runner audits each parsed record's map through the resolver and folds the drift diagnostics into the Quality Report's new parse section — the "drift diagnostics in the Quality Report" S2 asks for. The naive language heuristic was likewise lifted out of the Source into latence_core.language so Source and Parse share it rather than fork it.

Rejected: page_map on a Parser-package type (dependency inversion, above); a separate out-of-band page-map file keyed by document id (breaks the "every Stage boundary is one validated versioned record" invariant and the checkpoint round-trip resume depends on).

A corrupt file yields a PARSE_ERROR record, not an exception

A real Parse Stage meets malformed input: a truncated PDF, an encrypted PDF, a .pdf that is actually an executable (type spoof). The walking skeleton's passthrough never failed, so nothing forced the question. Letting a parse error propagate would abort the whole run on one bad document in a folder of thousands — the opposite of enterprise-grade.

Decision. The document Parser catches every recoverable failure and emits a DocumentRecord with disposition = PARSE_ERROR, an error reason, empty content, and no page_map — Provenance intact for audit. The run continues; the Quality Report counts parse_errors. A contract validator enforces the invariant both ways (a PARSE_ERROR must carry a reason; a PARSED record must not), so the disposition can never be set inconsistently. This is distinct from Quarantine (CONTEXT): Quarantine is Screening's disposition for dangerous data removed from the pipeline; PARSE_ERROR is Parse's disposition for data it could not decode. Screening (intake/content) is a later slice; this is the narrow, Parse-local failure path.

The type-spoof guard (reject a .pdf lacking the %PDF- signature before handing bytes to the reader) is a defense-in-depth check, not a substitute for the S-slice Intake Screening that will catch malware / zip bombs / oversized files ahead of Parse.

Rejected: raising and letting the Runner decide (couples Parser error semantics into the Runner and loses the per-document audit record); silently dropping the document (un-auditable — the Quality Report could not report what failed and why).