bodo-ai/Bodo

High Performance Scalable Data Processing in Python and SQL

What it solves

Bodo DataFrames addresses the performance and scalability bottlenecks of Python data processing, specifically for large-scale AI/ML workloads. It eliminates the runtime overheads found in traditional distributed frameworks (like Spark or Dask) and allows Pandas code to scale to larger-than-memory datasets without requiring a complete rewrite of the codebase.

How it works

It acts as a drop-in replacement for Pandas by replacing the import statement. Under the hood, it uses a C++ runtime and Message Passing Interface (MPI) technology to achieve true parallel execution. It also employs a just-in-time (JIT) compiler to accelerate custom transformations and native support for NumPy and Scikit-learn, alongside an integrated SQL engine and streaming execution for handling massive datasets.

Who it’s for

Data scientists and AI engineers who use Pandas for data preparation and feature engineering but need HPC-grade performance and the ability to scale their workloads across large cloud clusters or high-core laptops.

Highlights

  • Drop-in Compatibility: Accelerates workloads by simply changing import pandas as pd to import bodo.pandas as pd.
  • HPC Performance: Leverages MPI to be significantly faster than Spark or Dask.
  • JIT Compilation: Accelerates custom functions and transformations using native support for Pandas, NumPy, and Scikit-learn.
  • Enterprise Data Integration: Built-in scalable I/O for Apache Iceberg, Snowflake, Parquet, CSV, and JSON.
  • Memory Efficiency: Supports streaming execution to process datasets that exceed available RAM.

Related

  • Project
    hosseinmoein/DataFrame

    A high-performance C++ library for in-memory tabular data analysis and transformation, providing a fast alternative to Pandas and Polars for large-scale statistical and ML datasets.

  • Project
    pandas-dev/pandas

    A powerful Python data analysis toolkit that provides fast, flexible, and expressive data structures for working with relational or labeled data.

  • Project
    mito-ds/mito

    A set of Jupyter extensions that accelerate Python data analysis by providing an interactive spreadsheet that generates Python code and integrated AI assistance.

  • Project
    OpenMDAO/OpenMDAO

    A high-performance computing platform for systems analysis and multidisciplinary optimization that enables the decomposition and parallel solving of complex models.

  • Dispatch
    Pi Coding Agent: How Minimalism Improves Performance and Reduces Cost

    Pi is a minimalist coding harness that reduces token overhead and increases performance by providing a thin, extensible interface between LLMs and the development environment.