apache/datafusion-python

What This Project Is

DataFusion Python is a library that gives Python programmers access to DataFusion, a fast in-memory query engine written in Rust that works with Apache Arrow data. Instead of writing your own data-processing logic, you can use SQL or DataFrame-style calls and let DataFusion handle the heavy lifting. Because it's a “binding” (a bridge between Python and the Rust engine), it's often used as a building block for other data tools — the README lists Dask SQL, DataFusion Ballista, and DataFusion Ray as projects that build on it.

What It Lets You Do

From the README, the main features are:

• Run SQL queries or use a DataFrame API against CSV, Parquet, and JSON files.
• Queries are automatically optimized by DataFusion's built-in query optimizer.
• Execute user-defined Python functions inside SQL queries.
• Exchange data with Pandas and other DataFrame libraries that support PyArrow.
• Serialize and deserialize query plans in Substrait format (a standard way to represent query plans).
• Experimental support for transpiling SQL to DataFrame calls for Polars, Pandas, and cuDF.

The README also points to a guide for tuning parallelism to maximize CPU usage, but doesn't describe the details here.

A Concrete Example (from the README)

The README shows a small script that queries a Parquet file of NYC taxi trip data, counts trips by passenger count, converts the result to a Pandas DataFrame, and saves a bar chart:

from datafusion import SessionContext

ctx = SessionContext()
ctx.register_parquet('taxi', 'yellow_tripdata_2021-01.parquet')

df = ctx.sql("select passenger_count, count(*) "
             "from taxi "
             "where passenger_count is not null "
             "group by passenger_count "
             "order by passenger_count")

pandas_df = df.to_pandas()
fig = pandas_df.plot(kind="bar", title="Trip Count by Number of Passengers").get_figure()
fig.savefig('chart.png')

Another example shows how to register a DataFrame as a view so you can query it with SQL. You create a context, build a DataFrame from a Python dictionary, filter it, register it with register_view, then run SQL against the view and collect results.

Configuration

The README shows that you can tune runtime settings (memory, disk) and session settings when creating a context. Examples include setting the default catalog and schema, controlling how many partitions to use, and toggling options like repartition joins or Parquet pruning. You can print a context to see its current configuration.

How to Install

Three ways are listed:

• uv: uv add datafusion
• Pip: pip install datafusion (or python -m pip install datafusion)
• Conda: conda install -c conda-forge datafusion

You can verify the install by running import datafusion and checking datafusion.__version__.

Using It with AI Coding Assistants

The project ships a skill file that teaches AI coding assistants how to write idiomatic DataFusion Python. You can install it with npx skills add apache/datafusion-python, or manually paste a reference line into your project's AGENTS.md / CLAUDE.md.

Development Setup (for Contributors)

The README explains how to set up a development environment using uv or pip, with Rust and cargo installed. It uses maturin (a tool for building Python packages from Rust). Protobuf or cmake must be installed for protobuf support. Test data comes from git submodules. The project also offers:

• pre-commit hooks for linting (optional but recommended; require docker and cmake)
• Scripts for running Python and Rust linters (ci/scripts directory)
• A skill for checking upstream coverage — it helps audit which features from the upstream Apache DataFusion Rust library are not yet exposed in these Python bindings. You can ask it to check areas like scalar functions, aggregate functions, window functions, dataframe, session context, FFI types, or all at once.

What's Not Covered Here

This explainer is based only on the README, so I haven't invented features beyond what it states. For deeper documentation, the README points to the online API docs and an examples folder.