quixio/quix-streams

Python Streaming DataFrames for Kafka

What it solves

Quix Streams simplifies the process of building real-time data pipelines and event-driven microservices using Apache Kafka. It removes the need for complex server-side clusters and Java wrappers, allowing developers to perform extract, transform, and load (ETL) operations on streaming data using pure Python.

How it works

The framework uses two primary components: the Application object, which manages the Kafka connection, message lifecycle, and scaling via consumer groups, and the StreamingDataFrame (SDF), a declarative pipeline used to process and transform incoming messages. The SDF allows users to apply transformations, filters, and windowing operations to data as it flows through the Kafka topics.

Who it’s for

It is designed for data engineers, Python developers, and ML engineers who need to build reliable, scalable, real-time data processing systems and AI/ML products based on Kafka streams.

Highlights

  • Pure Python: No Java wrappers or cross-language debugging required.
  • Streaming DataFrame API: A tabular data processing approach for building pipelines.
  • Exactly-once Processing: Guaranteed via Kafka transactions to ensure data reliability.
  • Fault-tolerant Stateful Operations: Supports stateful processing with automatic checkpointing for resiliency.
  • Broad Integration: Includes a Serializers API for JSON, Avro, and Protobuf, and a dedicated API for custom sources and sinks.
  • Streaming Joins and Windowing: Built-in operators for complex event processing tasks like grouping and branching.

Related

  • Project
    kedro-org/kedro

    A Python framework for creating production-ready data engineering and data science pipelines using software engineering best practices.

  • Project
    Avaiga/taipy

    A Python framework for data scientists and ML engineers to build and deploy production-ready AI and data-driven web applications without needing frontend expertise.

  • Project
    pathwaycom/pathway

    Pathway is a Python‑first, Rust‑backed framework for building batch‑or‑stream data pipelines. It offers connectors to many sources, stateful operators, persistence, and a dedicated LLM toolkit (wrappers, parsers, vector index) that makes real‑time RAG and other LLM applications easy to write and deploy via Docker or Kubernetes.

  • Project
    timeplus-io/proton

    A high-performance C++ SQL pipeline engine for real-time stream processing and analytics, serving as a lightweight alternative to Apache Flink and ksqlDB.

  • Project
    pathwaycom/pathway-benchmarks

    A suite of reproducible benchmarks comparing the Pathway Live Data Framework's performance in streaming and batch data processing against Flink, Spark, and Kafka Streams.