elixir-explorer/explorer

Series (one-dimensional) and dataframes (two-dimensional) for fast and elegant data exploration in Elixir

What it solves

Explorer provides fast data exploration and manipulation capabilities for the Elixir language, bringing series (1D) and dataframes (2D) to the ecosystem. It allows developers to handle large datasets efficiently without leaving the Elixir environment.

How it works

Explorer uses a pluggable backend system, with its default backend based on NIF bindings to the Polars library, ensuring high-performance data processing. It implements an API influenced by Tidy Data and dplyr, providing a set of constrained "verbs" for data manipulation. The library supports various data formats including CSV, Parquet, NDJSON, and Arrow IPC, and can integrate with external databases via ADBC or connect directly to S3 storage.

Who it’s for

It is designed for Elixir developers who need to perform data analysis, data cleaning, and exploration of structured data within their applications or in interactive environments like Livebook.

Highlights

  • High Performance: Built on the Polars library for blazing-fast execution.
  • Linguistic Elegance: Combines the speed of Polars with an "Elixir-ish" immutable API.
  • Broad Format Support: Native support for Parquet, CSV, NDJSON, and Arrow IPC.
  • Flexible Integration: Connects to S3 and external databases via ADBC.
  • Strong Typing: Supports a wide range of simply typed series including binaries, booleans, dates, and various integer/float sizes.

Related

  • Project
    elixir-nx/axon

    A neural network framework for Elixir that provides high-level APIs for model creation and training, powered by Nx for numerical acceleration.

  • Project
    elixir-nx/nx

    A numerical computing library for Elixir that provides multi-dimensional arrays and high-performance backends like XLA and LibTorch for machine learning.

  • Project
    NVIDIA/cudf

    A GPU-accelerated DataFrame library for tabular data processing that provides a pandas-compatible API and zero-code change acceleration for pandas, Polars, and Dask.

  • Project
    brainlid/langchain

    An Elixir framework designed to help developers build data-aware and agentic applications by chaining LLMs with external tools, data, and services.

  • Project
    hosseinmoein/DataFrame

    A high-performance C++ library for in-memory tabular data analysis and transformation, providing a fast alternative to Pandas and Polars for large-scale statistical and ML datasets.