Skip to content

Latence Framework

Turn messy, siloed enterprise data into high-quality, AI-ready data — model-agnostic and cloud-agnostic.

Latence is a pipeline framework that takes documents from where they live (a folder, an object store, a source system) all the way to two portable, AI-ready deliverables: a RAG-ready corpus and a knowledge graph, both written as files (ADR-0017). It defines the stack and orchestration contracts; the concrete models and infrastructure are the adopter's choice.

  • Model-agnostic. The framework never names a model. It names Capabilities — narrow, typed interfaces like Parser, EntityExtractor, PIIDetector — and lets Providers (an in-process model, a remote endpoint, a cloud service) fulfil them as plugins. See Capability protocols with provider plugins.
  • Cloud-agnostic. Storage is an fsspec seam; the same Pipeline runs on a laptop or as an Airflow DAG through a swappable Runner. Two Runners ship today — the built-in local one and latence-runner-airflow; other substrates (Databricks Workflows, plain containers) are seam-compatible follow-ons, not shipped adapters.
  • CPU-first, offline by default. Every Stage ships a deterministic, dependency-light reference Provider that runs on a laptop with no GPU and no network (ADR-0007). GPU serving is an upgrade path, not a prerequisite.
  • Honest about licensing. Reference Providers ship permissive weights only; restricted-license models are opt-in (ADR-0012). Every Provider declares a ProviderProfile with its verified weights-and-code license, compute, and footprint.

The pipeline at a glance

flowchart LR Source --> Intake["Intake Screening"] --> Parse --> Chunk --> Content["Content Screening"] Content --> Extract["Entity + Relation Extraction"] Extract --> Redaction Redaction --> Profiling Profiling --> Disambiguation Disambiguation --> Graph["Graph Assembly"] Graph -. optional .-> Completion["Graph Completion"] Graph --> Export Completion --> Export Export --> Corpus["RAG corpus (files)"] Export --> KG["Knowledge graph (files)"]

That is the spine the blessed GPU stack declares (stacks/gpu-sota.yaml). Three further Stages are optional and inert unless a stack declares them — Schema Induction (ADR-0038), Type Consolidation (ADR-0054) and Context Enrichment (ADR-0039) — and embeddings are an opt-in Export augmentation rather than a Stage of their own (Embedders). Pipeline Stages documents each of them, with what it consumes and produces.

A Pipeline is a DAG of Stages over a set of input documents, declared once and executable by any Runner. Every record at every Stage carries immutable Provenance back to its source, and each run emits a first-class Quality Report.

Find your reading path

Four kinds of reader arrive here. Each path below is ordered — follow it top to bottom and you should not need to hunt for anything else.

You want to know whether the claims hold before spending a day on it.

  1. Messy data to knowledge-graph retrieval — a deep dive — the long-form argument: what happens at each step, and why fusing a knowledge graph into retrieval answers questions dense and lexical search structurally cannot.
  2. Concepts & Architecture — the five load-bearing concepts (Capability, Provider, Stage, Runner, Storage) and where the seams are.
  3. Pipeline Stages — the contract of every Stage, Source through Export.
  4. Public benchmark results — measured numbers, each carrying its dataset, hardware and measurement date. A Stage with no section there has not been measured; that is stated, never implied by silence.
  5. Provider catalog — every shipped Provider with its compute, footprint and verified weights-and-code license.
  6. Threat model · Continuity & security process · SBOM & supply chain — the procurement answers, each claim pointing at the artifact in the repository that proves it.

You want a corpus and a knowledge graph out of a folder of documents.

  1. Getting Started — install, latence setup, latence process, and what lands in the output directory.
  2. Run & validate a stack — the blessed stacks and latence stack validate.
  3. Read the results — the run console — one command, a localhost page, a quality verdict a non-engineer can act on.
  4. Tune a stack per device — latence tune profiles the actual box and keeps only settings that do not change output.
  5. Deploy (Compose · Helm · air-gapped) · Observability · Secrets handling — what production asks for.
  6. Going deeper: the fresh-pod walkthrough drives the full GPU stack stage by stage, and Run a bake-off picks a Provider on evidence rather than reputation.

You want the framework to use your model, endpoint or cloud service.

  1. Concepts & Architecture — what a Capability is, and why a Provider needs no base class and no import of core to satisfy one.
  2. Authoring a Provider — the end-to-end path: implement one method, declare a profile, register an entry point.
  3. Writing an adapter with AdapterBase — the optional helper that factors out device selection, batching and error mapping.
  4. ProviderProfile & device handling — the typed descriptor the framework routes device selection from.
  5. Passing Provider Conformance — the C1–C6 checklist that is the definition of enterprise-ready here, and the gate every adapter package passes.
  6. API Reference — the contracts, protocols and error taxonomy you program against, autodoc'd from the source.

You want to change the framework itself, or to understand why it is shaped this way.

  1. Decision Log (ADRs) — the authoritative source. Every seam has a record with the context, the choice, and the alternatives rejected.
  2. Releasing — one version across every publishable distribution, and the gate that must be green before a tag.
  3. Continuity & security process — how vulnerabilities are handled, and what happens if the maintainer stops.
  4. CONTRIBUTING.md in the repository root — the local dev loop, the merge gate, and the doc-truth tests that fail the build when a page drifts from the code.

Browse by section

  • Get started Install, run, and read the results.

  • Concepts & architecture Capability / Provider / Stage / Runner / Storage, and every Stage's contract.

  • Providers The catalog, the authoring guide, and the conformance gate.

  • Guides Operate, tune, deploy, benchmark, and feed a retrieval engine.

  • API reference Contracts, protocols, Runner, Storage and errors, autodoc'd from source.

  • Decision log The 65 architecture decisions behind the framework.

What the repository actually contains
Packages 33 distributions under packages/ — one thin core plus one package per Provider family (ADR-0016).
Decisions 65 architecture decision records in docs/adr/, numbered 0001–0066 (0040 was never issued).
Runners Two shipped: the built-in local Runner and latence-runner-airflow.
Deliverables A RAG corpus and a knowledge graph, as files — Parquet, JSONL, TTL, GraphML (ADR-0017).

Install

Nothing is published to PyPI — the distribution channel is the signed GitHub Release (ADR-0062, whose rationale is superseded and whose decision is reopened, not reversed, by ADR-0063). You install from a clone:

git clone https://github.com/ddickmann/latence
cd latence
uv sync

# Providers are separate packages — add only what a stack needs, by path, e.g.:
uv pip install -e packages/latence-parser-pdfplumber   # CPU PDF parser
uv pip install -e packages/latence-ner-gliner          # learned zero-shot NER

uv sync installs the dependency-light core — contracts, Capability protocols, Pipeline, local Runner, Storage and the latence CLI — plus the CPU-first, offline default set. The core has near-zero dependencies (ADR-0016): pydantic, fsspec, pyarrow, pyyaml, typer. Heavy model stacks live in their own Provider packages, so the core stays lean. Step-by-step: the quickstart.

Project status

Latence is a personally-owned, open-source framework licensed under the Apache License 2.0; commercial use is granted by the licence and needs no agreement (COMMERCIAL.md). Ownership and the personally-owned posture are unchanged (ADR-0001). See the README for the marketing overview, and the Decision Log for the reasoning behind every seam.