lance-format/lance
Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
What it solves
Lance provides a high-performance, open lakehouse format specifically designed for multimodal AI. It addresses the limitations of traditional SQL-centric lakehouse formats (like Parquet or Iceberg) which struggle with vector search, fast random access for ML training, and the efficient storage of multimodal data like images and video.
How it works
Lance implements a file format, table format, and catalog specification that can be built on top of object storage. It allows for the storage of embeddings, text, and raw multimodal blobs (images, audio, video) in a single unified format. It supports hybrid search—combining vector similarity search, full-text search (BM25), and SQL analytics—and provides zero-copy versioning with ACID transactions for data management.
Who it’s for
It is designed for AI/ML engineers who need to build search engines, feature stores, or large-scale ML training pipelines that require high-performance IO and efficient random access to multimodal datasets.
Highlights
- Hybrid Search: Combines vector, full-text, and SQL search on a single dataset.
- High-Speed Random Access: Up to 100x faster random access than Parquet or Iceberg.
- Multimodal Support: Native storage for images, videos, and audio with lazy loading.
- Data Evolution: Efficiently add columns for feature engineering without rewriting the entire table.
- Zero-Copy Versioning: Built-in ACID transactions, time travel, and branching.
- Broad Ecosystem: Integrates with Apache Arrow, Pandas, Polars, DuckDB, Ray, and Spark.