bytewax/bytewax

Python Stream Processing

What it solves

Bytewax provides a way to build stateful stream processing pipelines in Python. It bridges the gap between the high-performance distributed processing capabilities of engines like Apache Flink or Spark and the accessibility of the Python ecosystem, allowing developers to use their favorite Python libraries for real-time data transformation and machine learning.

How it works

Bytewax uses a dataflow computational model where users define a graph of operators and connectors. Data flows from an Input (such as Kafka or S3), through Operators (stateless transformations like map and filter, or stateful ones like reduce and fold_window), and finally to an Output (databases or message queues). The underlying engine is built in Rust for performance and distribution, while the interface is Python-first.

Who it’s for

It is designed for developers and data engineers who need to process event-driven data in real-time, scale from local development to Kubernetes clusters, and integrate online machine learning workloads into their data pipelines.

Highlights

  • Python-First Interface: Uses standard Python libraries and tooling for pipeline definition.
  • Stateful Processing: Automatically maintains and recovers state, supporting event-time windowing and fault tolerance.
  • Distributed Scaling: Scales horizontally across multiple workers and nodes using waxctl for Kubernetes deployments.
  • Rich Connector Ecosystem: Built-in support for Kafka, filesystems, WebSockets, and S3.

Related

  • Project
    apple-aiml-research/ml-hypersim

    Hypersim is a large synthetic indoor dataset (≈77 k HDR images, 1.9 TB) with per‑pixel geometry, semantics, and material channels, plus a V‑Ray‑based toolkit for generating and editing similar data. It’s designed for training and evaluating scene‑understanding models (depth, normals, segmentation, intrinsic image decomposition) and includes a ready‑made train/val/test split.

  • Project
    matrixorigin/matrixone

    MatrixOne is an AI-native, MySQL-compatible HTAP database that consolidates transactional, analytical, and vector search workloads while introducing Git-like version control for data.

  • Project
    Syngnat/GoNavi

    GoNavi is a lightweight, native‑binary database GUI built with Go + Wails and React. It supports dozens of data sources, offers a Monaco‑based SQL editor, virtualised result grids, and schema‑aware AI assistants (OpenAI, Gemini, Claude, etc.) that can generate or optimise queries. The app exposes an MCP HTTP interface for external agents, runs in ~20‑26 MB installers, and is Apache‑2.0 licensed.

  • Project
    TileDB-Inc/TileDB

    TileDB is a universal storage engine for dense and sparse multi-dimensional arrays, enabling efficient storage and access for complex data science workloads.