quiltdata/quilt

Quilt is a Scientific Data Management Platform on AWS that helps teams and AI find, trust, and reuse data through deeply versioned, context-rich data packages.

What it solves

Quilt addresses the challenge of keeping scientific data usable over time. It prevents the loss of context, structure, and lineage that typically occurs as teams and workflows evolve, which often slows down AI projects and forces data teams into manual support roles.

How it works

Quilt transforms cloud data into durable, searchable, and reusable "packages." Each package bundles data together with its metadata, documentation, and version history. Built on AWS, it allows organizations to manage data in place without requiring disruptive migrations.

Who it’s for

It is designed for scientific organizations, data scientists, and bioinformatics workloads where data reproducibility and trust are critical.

Highlights

  • Deep Versioning: Captures version history and lineage for data packages to ensure reproducibility.
  • AWS Native: Works with data in place on AWS, avoiding the need for rigid migrations.
  • Python SDK & CLI: Provides tools for creating, installing, and uploading packages to S3.
  • Hybrid Model: Offers both an open-source SDK/CLI and an enterprise platform for advanced search and visualization.

Related

  • Project
    datachain-ai/datachain

    DataChain is a Python library that turns files in cloud storage (S3, GCS, Azure) into versioned, typed datasets stored in a lightweight SQLite‑based “Dataset DB”. It provides a fast query engine (filters, joins, vector similarity) that runs on metadata without moving the raw files, supports checkpoint‑aware pipelines, and integrates with LLM agents (Claude, Cursor, Codex, Copilot, Pi) via an optional “skill”. A generated markdown Knowledge Base makes the data readable by both humans and agents. Distributed execution and sharing are offered through DataChain Studio.

  • Project
    AgnostiqHQ/covalent

    A Python library that abstracts infrastructure management, allowing AI/ML and scientific research code to run on any cloud or on-prem cluster with a single line of code.

  • Project
    qiantongtech/qData

    An open-source data middle platform for enterprise data governance and development, featuring ETL, metadata management, and AI-powered Text2SQL data Q&A.

  • Project
    treeverse/dvc

    A command line tool and VS Code extension for versioning data and ML models, tracking experiments, and managing reproducible ML pipelines.

  • Project
    InfuseAI/ArtiVC

    A command-line tool for data versioning on cloud storage that allows users to snapshot and switch between data versions without requiring a separate server.