Skip to content

One schema, dual serialization (JSONL + Parquet/Arrow)

Data contracts are defined exactly once as versioned Pydantic models (published also as JSON Schema for non-Python consumers); Provenance and Classification are required fields validated at every Stage boundary, so records cannot silently lose their source or attributes — the failure mode the private stack suffered with implicit per-stage JSONL schemas duplicated between SDK and workers. Physical serialization is pluggable behind the contract with two built-ins: JSONL (streaming, checkpoints, laptop demo) and Parquet/Arrow (lakehouse handoff — Databricks/Snowflake clients read Stage outputs natively). Rejected: JSONL-only (conversion tax at every lakehouse boundary), Arrow-native (hostile to nested per-document records and to simple Provider authors), and a Protobuf/Avro schema registry (heavy for a solo-authored v1, alienates the Python-first adopter).