Skip to content

Parse consumes a raw ParserInput, not a pre-decoded DocumentRecord

The S1 walking skeleton over-fit the Parse seam to its own passthrough Provider. Source.produce yielded a DocumentRecord — a parsed record, markdown content already decoded — and Parser.parse consumed Iterable[DocumentRecord] and handed it straight through. That types cleanly only because the reference Parser is a plain-text passthrough; a real Parser consumes BYTES (a PDF, a scanned image, an office file) and produces the markdown. Wiring the Source to decode first inverts the responsibility: decoding-to-text is the Parser's defining job, and a PDF Parser has no str content to receive. The seam described a passthrough, not the Stage.

Decision. The input side of Parse is its own contract, ParserInput, distinct from the parsed DocumentRecord it produces:

  • A ParserInput is a raw document: provenance + classification stamped by the Source, plus content: bytes | str that is the undecoded source content exactly as read from Storage (bytes for binary formats; str allowed for an already-textual Source), plus a config: dict[str, Any] of per-input Parser hints (a forced encoding, an OCR language) so a Source or Runner can steer a Parser without minting a new contract. It is versioned (schema_version) like every other contract.
  • Source.produce(storage) -> Iterator[ParserInput] yields raw documents. The Source stamps Provenance and a naive Classification, decodes a view of the bytes only for its own language heuristic, and hands the raw bytes downstream — it never claims to have parsed anything.
  • Parser.parse(inputs: Iterable[ParserInput]) -> Iterator[DocumentRecord] owns the bytes→markdown turn. The plain-text reference Provider decodes here (honouring a per-input encoding hint), fills in the character span from the decoded length, and inherits Provenance/Classification losslessly. This is the exact input shape a Textract/Azure-DI/OCR Provider needs, so a second Parser is a drop-in, not a re-plumb.
  • The Runner's per-Stage carrier is ParserInput | DocumentRecord; a Source's checkpoint serializes ParserInput (via model_dump_json, which round-trips raw bytes losslessly), a Parser/Export's serializes DocumentRecord, and a capability that receives the wrong carrier (e.g. a Source wired straight into an Export) fails with a clear typed error rather than a downstream AttributeError.

This strengthens ADR-0004 (Capability protocols): the Capability seam is now typed against the data the Stage actually consumes and the data it produces as two contracts, so the model-agnosticity promise holds for a real bytes-consuming Provider, not only the demo passthrough. It respects ADR-0006 (one schema, dual serialization): ParserInput is another versioned Pydantic contract, and Provenance/Classification remain required at the seam.

Rejected: keeping DocumentRecord as the Parse input (the over-fit above — forces Sources to decode, which they cannot do for binary formats, and gives PDF Parsers a nonsensical pre-decoded str); a single union-typed record with a "is-it-parsed-yet" flag (a mode bit on one contract invites Providers to read the wrong half and defers the type error to runtime); and pushing raw content through an out-of-band Storage handle instead of the contract (breaks the "every Stage boundary is a validated versioned record" invariant and loses the checkpoint round-trip that resume depends on).

S13 build note — binary-safe ParserInput checkpoint serialization. The claim above that model_dump_json "round-trips raw bytes losslessly" held only for byte sequences that happen to be valid UTF-8. Pydantic's default bytes→JSON encoding is a UTF-8 decode, which raises on any non-UTF-8 sequence — so the first genuinely binary source document threaded through the Source/Intake checkpoint (a PDF is mostly ASCII and survived; a ZIP archive's deflate stream does not) tore the Runner's checkpoint mid-run. The S13 demo, whose whole point is ingesting a messy corpus including a Screened ZIP bomb, surfaced this. Fix: ParserInput now sets ser_json_bytes="base64" / val_json_bytes="base64", so content round-trips any bytes losslessly (model_validate_json yields the identical bytes). This is a serialization-encoding change only — the model fields, the inter-Stage contract, and its schema_version are unchanged, and every existing round-trip test still passes — so the checkpoint/resume invariant now holds for arbitrary binary source content, as this ADR always intended.