narwhals-dev/narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

What it solves

Narwhals is a lightweight compatibility layer that allows developers to write dataframe-agnostic code. It eliminates the need for a library author to manually implement support for multiple dataframe libraries (like pandas, Polars, cuDF, and PyArrow) by providing a single, unified API.

How it works

Narwhals wraps native dataframe objects using from_native and allows the user to perform operations using a subset of the Polars API. Once the operations are defined, to_native is used to return the object to its original library format. This process ensures that computation remains native to the input library, maintaining performance and overhead is negligible.

Who it’s for

It is primarily designed for maintainers of libraries that consume dataframes, enabling them to support multiple backends without adding them as dependencies.

Highlights

  • Broad Compatibility: Supports cuDF, Modin, pandas, Polars, and PyArrow natively, with lazy-only support for Dask, DuckDB, Ibis, PySpark, and others via plugins.
  • Zero Dependencies: The library does not depend on the dataframe libraries it supports, keeping the host library lightweight.
  • Static Typing: Provides full static typing to assist IDEs.
  • Performance: Maintains negligible overhead while ensuring 100% branch coverage.
  • Backwards Compatibility: Follows a strict policy to ensure API stability.

Related

  • Project
    NVIDIA/cudf

    GPU‑accelerated DataFrame library that mirrors pandas, Polars, and Dask APIs, enabling fast tabular data cleaning and aggregation for AI/ML workflows.

  • Project
    joaopauloschuler/neural-api

    A native Pascal deep-learning library that enables running LLMs, transformers, and other AI models as single native binaries without Python or CUDA.

  • Project
    pandas-dev/pandas

    Pandas is a foundational Python library for data manipulation and analysis, providing high-performance, flexible data structures to handle labeled and relational data.

  • Project
    Luos-io/luos_engine

    An open-source lightweight library for managing hardware products as a collection of independent software features, enabling faster development and reuse across microcontrollers.

  • Project
    AgnostiqHQ/covalent

    A Python library that abstracts infrastructure management, allowing AI/ML and scientific research code to run on any cloud or on-prem cluster with a single line of code.