datachain-ai/datachain

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

DataChain – A Python library for versioned, queryable datasets on unstructured data

What it is

  • A Python package that treats files stored in S3, GCS, Azure, or local disks as typed, version‑controlled datasets.
  • It builds a lightweight SQLite‑backed Dataset DB that stores schemas (via Pydantic), file pointers, lineage, and even vector embeddings, enabling sub‑second filtering, joins, aggregations, and similarity search without moving the raw files.
  • An optional Knowledge Base layer exports this metadata as markdown for humans and LLM agents, and a Skill can install a data‑aware agent harness for Claude, Cursor, Codex, GitHub Copilot, or Pi.

Key components

Component Role
Compute Engine Parallel Python over files (async I/O, checkpointing, incremental updates).
Dataset DB Stores Pydantic schemas, versioned dataset definitions, and vectors; queries run directly against SQLite.
Knowledge Base Auto‑generated markdown describing datasets, useful for agents and documentation.
Agent Harness / Skill Exposes read_storage, map, save, etc., to LLM agents so they can reason about and manipulate data.
Studio (cloud service) Optional hosted UI for shared dataset registry, distributed compute, access control, and lineage visualisation.

Typical workflow

  1. Read storage – dc.read_storage('s3://bucket/**/*.jpg', anon=True, delta=True) creates a lazy collection of files.
  2. Define a schema – Write a Pydantic model; the return type of a mapping function becomes the dataset schema.
  3. Map / transform – Apply a Python UDF (.map(info=get_info)) to each file; DataChain handles batching, checkpointing, and only re‑processes new/changed files.
  4. Save – .save('pets_images') registers the result as a versioned dataset (pets_images@1.0.0).
  5. Query – Use the fluent API (filter, mutate, order_by, limit) to run fast SQL‑like operations or vector similarity searches directly on the Dataset DB.
  6. Knowledge Base – Run dc-knowledge generation to produce markdown that LLM agents can read.
  7. Agent‑driven pipelines – With the installed skill, an LLM (e.g., Claude Code) can generate the whole pipeline from a natural‑language prompt.

Why it matters

  • No data copying – Datasets are just metadata and pointers; raw files stay in your cloud bucket.
  • Versioning & reproducibility – Every pipeline step is a named, immutable dataset version, making it easy to roll back or reuse results.
  • Scalable queries – SQLite‑based engine can filter and join millions of rows in milliseconds, far beyond what in‑memory pandas can handle.
  • Agent‑ready – The skill bridges LLM agents with your data, letting them understand dataset schemas, perform joins, and generate code automatically.

Installation

pip install datachain
# optional agent skill
# datachain skill install --target claude   # also cursor, codex, copilot, pi

Example (find dogs similar to a reference image, filtered by breed, mask, and size) – the README shows a full prompt‑to‑result flow where Claude Code decomposes the task, builds intermediate datasets, and returns a ranked list of matches.

Target audience

  • Data engineers and ML teams that need reproducible pipelines over large, unstructured media collections.
  • AI developers building LLM‑driven agents that must interact with real data without moving it.
  • Organizations that store data in cloud object stores and want a lightweight, version‑controlled data catalog.

Links

Related

  • Project
    OpenDCAI/DataFlow

    DataFlow is an open‑source Python framework that lets you build, run, and share low‑code pipelines for generating, cleaning, and evaluating LLM training data. It provides a library of reusable operators, a visual WebUI, an AI‑assistant for auto‑creating pipelines, and a Ray‑based distributed execution layer.

  • Project
    pdpipe/pdpipe

    A library for creating easy, chainable preprocessing pipelines for pandas DataFrames to streamline data cleaning and feature engineering.

  • Project
    roboflow/roboflow-python

    Roboflow‑python is the official Python client for the Roboflow computer‑vision platform. It lets you programmatically create workspaces/projects, upload and version datasets, trigger model training, and run inference on hosted or self‑hosted models. Install via `pip install roboflow` (or the lightweight `roboflow-slim`), authenticate with an API key or login, and use a simple object hierarchy (workspace → project → version) to manage the full CV lifecycle.

  • Project
    Affitor/affiliate-skills

    The repository provides a collection of 52 AI‑driven “skills” that turn any LLM into a full‑stack affiliate‑marketing assistant, covering research, content creation, landing‑page building, analytics and automation, with a closed‑loop flywheel and a tiny CLI for testing.

  • Project
    Trampoline-AI/predict-rlm

    predict-rlm is a Python library that gives large language models a sandboxed, stateful runtime. Models write and execute Python code, keep mutable state, call focused sub‑models, and use tools/skills (e.g., PDF handling, spreadsheets). The framework records a full trace for observability and can feed those traces to optimisation tools like GEPA. It’s aimed at complex, multi‑step workflows such as document analysis, audit, or spreadsheet automation, and supports optional Docker sandboxing and rich callbacks.