apache/arrow

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics

What it solves

Apache Arrow solves the problem of slow and inefficient data movement between different software systems. It provides a universal, standardized way to store and share data in memory, allowing different applications and programming languages to work together without the performance penalty of constantly converting data formats.

How it works

Arrow uses a "columnar" format, which organizes data by columns rather than rows. This structure is highly efficient for modern computing hardware. It includes a set of tools for memory management, such as "zero-copy" sharing, which allows different processes to access the same data without needing to copy it. It also provides standardized communication protocols (like Arrow Flight) and database connectivity (ADBC) to move this data across networks or between different software environments seamlessly.

Who it’s for

It is designed for developers and data engineers building high-performance data systems, databases, query engines, or analytical applications that need to process large volumes of data across multiple programming languages or distributed services.

Highlights

  • Universal columnar format that supports both flat and nested data types.
  • High-performance memory management with zero-copy capabilities.
  • Broad multi-language support including C++, Python, Rust, Java, Go, R, and more.
  • Built-in support for remote procedure calls (RPC) and interprocess communication (IPC).
  • Includes Gandiva, an expression compiler that uses LLVM to speed up data processing.
  • Provides standardized database connectivity (ADBC) and file format support (like Parquet and CSV).

Related

  • Project
    argoproj/argo-workflows

    A container-native workflow engine for Kubernetes that orchestrates parallel jobs, specifically optimized for machine learning pipelines and data processing.

  • Project
    cvat-ai/cvat

    An open-source, self-hosted platform for annotating images, video, audio, and 3D data to create high-quality datasets for AI training.

  • Project
    neo4j/neo4j

    Neo4j is a graph database that stores data as nodes and relationships instead of tables. It's available as a managed cloud service (Aura), a standalone server, or a desktop app. The open-source Community Edition is GPLv3; the Enterprise Edition has closed-source parts not in this repo. You can build it from source with Maven and Java 21, run it locally on port 7474, and extend it with your own code (after signing a CLA).

  • Project
    RyanCodrai/turbovec

    An in-process vector search library for Python and Rust that uses TurboQuant to compress embeddings by up to 15.5x, enabling fast, memory-efficient similarity search without the need for a dedicated database server.