laminlabs/lamindb

What it solves

LaminDB provides a unified system for managing multimodal AI datasets, specifically addressing the lack of traceability and governance in research workflows. It prevents the "black box" problem where results cannot be trusted because the exact data, code, and environment used to produce them are unknown. It is particularly designed for high-compliance environments like life sciences (GxP compliance) where end-to-end lineage is mandatory.

How it works

LaminDB uses a scalable lakehouse architecture that separates metadata (stored in SQLite or Postgres) from the actual data (stored in open formats like Parquet and Zarr). It integrates files, tables, arrays, and ontologies into a single system.

Key mechanisms include:

• Lineage Tracking: Automatically records the relationship between source code, compute environments, inputs, and outputs using ln.track().
• Versioning: Co-versions code and datasets, allowing users to track changes and revert to previous states.
• Branching: Implements a git-like branching and merging system to isolate data changes.
• Schema Validation: Uses features and registries to validate the content of DataFrames and biological data formats (like AnnData).

Who it’s for

It is built for scientists and engineers in biotech companies, research hospitals, and academic institutions who manage massive, multimodal datasets (ranging from proteins and genes to biosamples) and require strict reproducibility.

Highlights

• Zero Lock-in: Uses open standards for metadata and storage.
• Multimodal Support: Native handling of biological formats, tables, and arrays.
• Agent-Ready: Includes skills for AI agents to automatically track their own sessions and plans.
• Distributed Federation: Allows zero-copy data transfer and lineage-aware sharing across different databases.
• Ontology Integration: Supports public biological ontologies via the bionty plugin.