lance-format/lance

What it solves

Lance is designed to bridge the gap between traditional data lakehouse formats and the specific needs of modern AI/ML workflows. While formats like Parquet or Iceberg are excellent for SQL analytics, they often struggle with the high-performance requirements of AI, such as fast random access for training, multimodal data handling, and integrated vector search. Lance provides a unified format that handles these tasks without requiring separate, specialized infrastructure.

How it works

Lance functions as an open lakehouse format that includes a file format, table format, and catalog specification. It operates directly on object storage (like S3, GCS, or Azure) or local disks. It uses efficient blob encoding and lazy loading to manage multimodal data—such as images, video, audio, and text—alongside embeddings. It supports ACID transactions and zero-copy versioning, allowing users to perform time travel and branching on their data without needing extra management systems.

Who it’s for

Lance is built for data scientists, ML engineers, and developers who need to manage large-scale multimodal datasets. It is particularly useful for those building search engines, feature stores, or training pipelines who want to combine vector similarity search, full-text search, and SQL analytics within a single, high-performance ecosystem.

Highlights

• Expressive hybrid search: Combines vector similarity, full-text (BM25), and SQL analytics with accelerated secondary indices.
• Lightning-fast random access: Offers performance up to 100x faster than Parquet or Iceberg for random access tasks.
• Native multimodal support: Efficiently stores and manages images, audio, video, and embeddings in a unified format.
• Data evolution: Allows adding columns and backfilling values without requiring full table rewrites.
• Rich ecosystem integration: Works natively with popular tools like Pandas, DuckDB, Polars, Apache Arrow, Spark, and Ray.