pola-rs/polars

What it solves

Polars solves the performance and memory limitations often encountered when working with large datasets in DataFrames. It provides a high-performance engine that can handle datasets larger than your available RAM through streaming, while optimizing queries automatically to ensure efficient execution.

How it works

Written in Rust, Polars uses a multi-threaded, vectorized execution engine that leverages modern CPU capabilities (SIMD). It supports both eager and lazy execution, allowing the engine to optimize your query plan before it runs. It uses the Apache Arrow format for zero-copy data sharing, ensuring high interoperability with other tools. You can compose complex data transformations using a powerful, expressive API, and even extend the engine with custom plugins.

Who it’s for

Polars is for data scientists, engineers, and analysts who need to process large-scale data efficiently. It is designed for users working in Python, Rust, Node.js, R, or SQL environments who require a fast, scalable alternative to traditional DataFrame libraries.

Highlights

• Extremely fast performance via multi-threaded and vectorized execution.
• Streaming engine allows processing datasets that are larger than your computer's RAM.
• Query optimization happens automatically for lazy execution plans.
• Multi-language support including Python, Rust, Node.js, R, and SQL.
• Optional GPU acceleration for supported NVIDIA hardware.
• Native extensibility through I/O and expression plugins.