lance-format/lance

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

What it solves

Lance provides a high-performance, open lakehouse format specifically designed for multimodal AI. It addresses the limitations of traditional SQL-centric lakehouse formats (like Parquet or Iceberg) which struggle with vector search, fast random access for ML training, and the efficient storage of multimodal data like images and video.

How it works

Lance implements a file format, table format, and catalog specification that can be built on top of object storage. It allows for the storage of embeddings, text, and raw multimodal blobs (images, audio, video) in a single unified format. It supports hybrid search—combining vector similarity search, full-text search (BM25), and SQL analytics—and provides zero-copy versioning with ACID transactions for data management.

Who it’s for

It is designed for AI/ML engineers who need to build search engines, feature stores, or large-scale ML training pipelines that require high-performance IO and efficient random access to multimodal datasets.

Highlights

  • Hybrid Search: Combines vector, full-text, and SQL search on a single dataset.
  • High-Speed Random Access: Up to 100x faster random access than Parquet or Iceberg.
  • Multimodal Support: Native storage for images, videos, and audio with lazy loading.
  • Data Evolution: Efficiently add columns for feature engineering without rewriting the entire table.
  • Zero-Copy Versioning: Built-in ACID transactions, time travel, and branching.
  • Broad Ecosystem: Integrates with Apache Arrow, Pandas, Polars, DuckDB, Ray, and Spark.

Related

  • Project
    lancedb/lancedb

    A multimodal AI lakehouse and vector database built on the Lance columnar format for fast, scalable search over petabytes of vectors and multimodal data.

  • Project
    activeloopai/deeplake

    Deep Lake is a serverless database for AI that stores and streams large-scale unstructured data and vectors, optimizing the workflow for deep learning training and LLM applications.

  • Project
    ModernRelay/omnigraph

    Omnigraph is a Rust‑based, lakehouse‑style graph database that stores data in the columnar Lance format on any S3‑compatible object store. It is built for fleets of AI agents: each agent can write to its own isolated branch, then submit changes for review and Git‑style merge. Queries fuse graph traversal, vector ANN, and full‑text search in one runtime, and Cedar policies enforce fine‑grained security on every mutation. The project ships a CLI, an Axum HTTP server, TypeScript SDKs, and a “MCP” bridge for LLM hosts, making it a full‑stack solution for agentic memory, company‑brain knowledge graphs, dev‑graph automation, and versioned ML data layers.

  • Project
    laminlabs/lamindb

    LaminDB is an open‑source, Git‑like data‑management system for AI/ML, especially multimodal life‑science data. It version‑controls files, tables, and array formats, tracks code and environment provenance, supports branching/merging, and offers ACID‑safe metadata stored in SQLite/Postgres. Integrated with Python/R tools (Polars, DuckDB), bio‑ontologies, and workflow managers, it enables traceable, FAIR‑compliant datasets for research and biotech.

  • Project
    lakesoul-io/LakeSoul

    LakeSoul is an open‑source lakehouse framework with a Rust‑native metadata and I/O core, offering ACID transactions, incremental upserts, automatic compaction, fine‑grained RBAC, and support for Spark, Flink, Presto, Ray, Daft, and Python‑based tools. It lets you store regular rows and multimodal/vector data (via the Vortex format) and provides real‑time CDC ingestion, snapshot queries, and seamless integration with AI/ML pipelines.