vortex-data/vortex

An extensible, state-of-the-art framework for columnar compression, and the fastest FOSS columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation.

What it solves

Vortex is a high-performance columnar file format and toolkit designed to optimize data processing, particularly for systems backed by object storage. It addresses the performance bottlenecks of existing formats like Apache Parquet, offering significantly faster random access reads, scans, and writes while maintaining similar compression ratios.

How it works

Vortex separates the logical schema (data types) from the physical layout (how data is stored). It uses a pluggable architecture for encoding, compression, and layout strategies, allowing it to be highly extensible. It is designed for zero-copy compatibility with Apache Arrow, meaning data can be transferred between Vortex and Arrow without expensive copying or parsing of metadata.

Who it’s for

This project is for developers building data systems, analytics engines, and high-performance computing applications that require efficient storage and storage-backed data processing.

Highlights

  • Extreme Performance: Up to 100x faster random access reads and 10-20x faster scans compared to modern Apache Parquet.
  • Extensible Architecture: Pluggable systems for encoding, types, and compression.
  • Arrow Integration: Zero-copy compatibility with Apache Arrow arrays.
  • Broad Integration: Supports integrations with DataFusion, DuckDB, Spark, Pandas, and Polars.
  • Neutral Governance: Managed as a Linux Foundation (LF AI & Data) project under the Apache-2.0 license.

Related

  • Project
    facebookincubator/nimble

    A columnar file format by Meta designed for large, wide datasets, optimizing storage and access for machine learning training tables and feature engineering.

  • Project
    zzet/gortex

    A high-performance code-intelligence engine that indexes codebases into a knowledge graph to provide AI agents with precise context, reducing token usage by up to 50x.

  • Project
    lance-format/lance

    An open lakehouse format for multimodal AI that provides high-performance vector search, random access, and native storage for images, video, and audio.

  • Project
    StarTrail-org/LEANN

    A lightweight vector database that uses graph-based selective recomputation to reduce storage by 97%, enabling local RAG on millions of personal documents.

  • Project
    questdb/questdb

    A low-latency, open-source time-series database that ingests millions of events per second and provides high-performance SQL queries over live and historical data.