Skip to content

Local Runner state is files-on-Storage, no database

The built-in local Runner persists all run state — run manifest, per-Stage checkpoints, resume metadata — as files on the same fsspec Storage as the data, guarded by a write-ahead log ported from dataset-intelligence. There is no database to stand up: the demo runs against a folder and the identical code resumes a half-finished run from S3. This removes the private stack's hard dependency on Supabase/Postgres for job state. Adopter-grade Runners (Databricks Workflows, Airflow) deliberately use their own platform's state and scheduling, not this — the local Runner's state model is not a control plane and does not try to be one. Rejected: embedded SQLite (does not work on object stores, splitting state across two substrates and breaking resume-from-anywhere), a pluggable Postgres StateStore in v1 (surface better deferred to a post-v1 plugin), and in-memory-only (discards the checkpoint/resume capability enterprises need for large-corpus reprocessing).