Skip to content

Python core, declarative-YAML Pipeline contract, Rust only for proven text hot paths

Pipeline declaration: A versioned YAML/JSON schema is the canonical Pipeline contract — Stages, the Provider bound to each Stage (plugin name + config), Storage URIs, Screening policy. A typed Python builder constructs and validates the same object and can emit the YAML; Runners consume the declaration. Config-as-data is what platform teams diff, review, and commit, and it makes the demo's one-line Provider swap literally true. Rejected: Python-builder-primary (weaker governance, swap doesn't read as declarative), YAML-only (loses typed/testable construction and per-tenant dynamism), and Hydra/OmegaConf (heavy opinionated dependency that leaks into every adopter's config, overkill for the fixed v1 spine).

Implementation language: Python is the core and the Provider language. Rationale: the entire model ecosystem (transformers, GLiNER, PaddleOCR, sentence-transformers, vLLM, pyarrow, fsspec, pydantic) is Python and Providers must call it in-process; the first adopter is a Python data scientist for whom a Provider must be pip install + subclass; the orchestration layer is IO/inference-bound, not CPU-bound (real compute is C++/CUDA inside the models); and Databricks/Spark, Airflow, and cloud SDKs are Python-native. A Rust core would impose a PyO3 boundary on every model call for no throughput gain and gut community contribution.

Rust is admitted ONLY as optional native acceleration for specific CPU-bound text hot paths — chunking, offset/page-alignment, corpus-scale PII scanning — always behind a Python API with a pure-Python fallback so first-run never requires a Rust toolchain. This mirrors the already-validated text_processing Rust wheel (with Python fallback) in the private superpod stack. Reversal cost is why this is recorded: choosing a Rust core later would be a rewrite, whereas adding a Rust extension behind an existing Python seam is incremental.