laminlabs/lamindb

LaminDB – Data‑management platform for multimodal AI workflows

What it is – LaminDB is an open‑source library (Python + R) that lets you treat any kind of scientific data – Parquet tables, Zarr arrays, AnnData objects, images, plain files, etc. – as artifacts stored in a version‑controlled “lakehouse”. It records where each artifact came from, which code (or AI agent) produced it, and any custom metadata you attach. Think of it as git for data, but with ACID‑guaranteed transactions, time‑travel, branching, and schema evolution.

Why it matters for AI – Modern AI pipelines (large‑language‑model agents, multimodal models, bio‑informatics workflows) juggle huge, heterogeneous datasets. Without traceability you can’t audit results, reproduce experiments, or safely merge contributions from many collaborators. LaminDB provides:

• Lineage tracking – every run, transform, and artifact is linked, so you can reconstruct exactly how a model’s output was generated.
• Lakehouse guarantees – ACID transactions, snapshot isolation, and versioned data for non‑tabular formats.
• Schema‑driven validation – define features (e.g. gc_content, experiment_date) and automatically validate DataFrames, AnnData, SpatialData, etc.
• Zero‑copy sharing – move references between databases without duplicating large files.
• Integration with existing tools – works directly with Polars, DuckDB, Nextflow, redun, Vitessce, and bio‑ontologies via the bionty plugin.

How you use it

1. Install – pip install lamindb (or the minimal lamindb-core).
2. Connect – lamin login && lamin connect account/name or initialise a local repo with lamin init.
3. Save data – ln.Artifact("sample.fasta", key="sample.fasta").save() or via the CLI lamin save ….
4. Track code – wrap scripts with ln.track() / ln.finish() or decorate functions with @ln.flow() to capture parameters, environment, and generated artifacts.
5. Query – ln.Artifact.filter(...).to_dataframe() lets you search by metadata, size, run, or custom features.
6. Branch & merge – use lamin switch -c my_branch and lamin merge just like git to isolate and integrate changes.

Who uses it – Teams at Pfizer, Altos Labs, academic consortia (scverse, Helmholtz Munich), and research hospitals (Harvard, MIT, ETH Zürich) manage petabyte‑scale multimodal datasets with LaminDB.

Key take‑aways

• Provides a distributed, language‑agnostic data catalog (SQLite/Postgres metadata, files on S3, GCP, local disk).
• Enables full reproducibility for AI/ML pipelines by coupling data, code, and compute environment.
• Designed for life‑science and multimodal AI but applicable to any domain needing trustworthy data governance.