datachain-ai/datachain

DataChain – a “context layer” for unstructured data

What it is – A Python library that turns files stored in S3, Google Cloud Storage, Azure Blob, or a local filesystem into typed, version‑controlled datasets that can be queried like a data‑warehouse. It adds a thin “data harness” that lets LLM‑powered agents (Claude, Cursor, Codex, Copilot, Pi) understand and operate on those datasets without moving the raw files.

Core pieces

Component	What it does
Compute Engine	Parallel, async Python over files; supports checkpointing, incremental (delta) runs, and distributed execution via DataChain Studio.
Dataset DB	SQLite‑backed store that records a Pydantic schema, version, lineage and file pointers for each dataset. Queries (filter, join, group‑by, vector search) run directly against this DB in sub‑second time, even on hundreds of millions of rows.
Knowledge Base (optional)	Auto‑generated markdown that describes every dataset and its schema; readable by humans and by LLM agents.
Agent Harness (optional)	A skill that plugs the three layers into LLM code‑generation agents so they can issue read_storage, map, save, etc., and retrieve results as if they were normal Python objects.

Typical workflow

1. Read raw files – dc.read_storage('s3://bucket/**/*.jpg', anon=True, delta=True) creates a lazy view of the files.
2. Define a schema – Write a Pydantic model (e.g., ImageInfo) that describes the columns you want.
3. Map / transform – Apply a Python UDF (map(info=get_info)) that returns the model; DataChain stores the results as a new named dataset (pets_images@1.0.0).
4. Save – .save('pets_images') registers the dataset in the Dataset DB, versioning it automatically.
5. Query – Use dc.read_dataset('pets_images').filter(...).to_pandas() or vector‑search helpers (dc.func.cosine_distance) to retrieve or rank rows.
6. Incremental runs – With delta=True only newly‑added files are processed; the version number bumps (@1.0.1, @1.0.2, …) while unchanged rows are skipped.
7. Agent‑driven pipelines – After installing the skill (datachain skill install --target claude), an LLM can be prompted to build the whole pipeline, generate the Knowledge Base, and answer follow‑up questions using the stored datasets.

Key features highlighted in the README

• Versioned, typed datasets – every pipeline step produces a reproducible, named dataset with a schema stored in SQLite.
• Fast, sub‑second queries – filters, joins, aggregations, and vector similarity search run directly on metadata without loading files.
• Checkpoint‑aware pipelines – crashes or bugs don’t force a full re‑run; the engine resumes from the last successful batch.
• Agent integration – a skill for Claude, Cursor, Codex, Copilot, and Pi lets LLMs read the Knowledge Base and call DataChain operations as if they were native functions.
• Scalable compute – local multi‑core async execution or distributed jobs via DataChain Studio (GPU clusters, on‑prem K8s, any cloud).
• Zero data movement – raw files stay in the original bucket; only lightweight metadata and optional embeddings are stored locally.
• Knowledge Base – markdown representation of datasets for human browsing (Obsidian‑compatible) and for agents to discover available data.

Installation & quick start

pip install datachain                     # core library
# optional LLM skill (example for Claude)
datachain skill install --target claude

A minimal “agent‑driven” example from the README:

1. Copy a reference image from S3.
2. Open Claude Code (or another supported agent).
3. Prompt it to find dogs similar to the image, filtered by breed, mask availability, and width.
4. The agent decomposes the request, runs embeddings, joins breed metadata, applies filters, and returns a ranked table – all using datasets that are automatically versioned and stored.

Studio – shared, cloud‑backed execution

• datachain auth login → authenticate to the hosted UI.
• datachain job run --workers 20 --cluster gpu-pool script.py submits a distributed job.
• The UI shows lineage graphs, dataset registry, access control, and supports large medical formats (DICOM, NIfTI, point clouds).

Who might use it

• Machine‑learning teams that need reproducible data pipelines over massive image/video/audio collections.
• Companies building LLM‑augmented agents that must query their own data without copying it to a separate vector store.
• Researchers who want fast, SQL‑like exploration of billions of file‑level metadata and embeddings.
• Any organization that wants versioned, typed datasets with built‑in lineage and incremental updates.

Where to learn more

• Full docs: https://docs.datachain.ai/
• Architecture diagram: docs/assets/harness.svg
• Community: GitHub issues, email support, Twitter @datachain_ai.

---

All statements are taken directly from the repository’s README; no additional features have been inferred.