quixio/quix-streams

quixio/quix‑streams – Python framework for real‑time data engineering on Kafka

What it is – An open‑source Python library that lets you build reliable, end‑to‑end streaming pipelines on top of Apache Kafka. It provides a Streaming DataFrame API and a lightweight Application object that handle Kafka connectivity, serialization, state‑management and exactly‑once guarantees, so you can focus on the transformation logic.

Key capabilities

• Pure‑Python implementation – no Java wrappers or cross‑language debugging.
• Streaming DataFrame API for declarative, tabular processing (apply, filter, group‑by, window, joins, etc.).
• Pluggable Sources and Sinks to connect arbitrary Kafka topics.
• Built‑in serializers for JSON, Avro, Protobuf and Schema Registry.
• Fault‑tolerant stateful operators with automatic checkpointing.
• Exactly‑once processing via Kafka transactions.
• Works with any Python‑based analytics or ML library, making it easy to embed model inference or training in a live stream.

Typical workflow

from quixstreams import Application

app = Application(broker_address="localhost:9092")

in_topic  = app.topic("temperature-celsius", value_deserializer="json")
out_topic = app.topic("temperature-alerts",  value_serializer="json")

sdf = app.dataframe(topic=in_topic)
# transform, filter, then write out
sdf = sdf.apply(lambda v: {"temperature_F": v["temperature"]*9/5 + 32})
        .filter(lambda v: v["temperature_F"] > 150)
        .to_topic(out_topic)

app.run()

The Application consumes, deserialises, runs the dataframe logic, produces results, and checkpoints state automatically.

Getting started

• Install via PyPI: python -m pip install quixstreams (or Conda).
• Requires Python 3.11+ and Kafka 0.10+.
• Full docs, quick‑start guide, and several tutorials (word‑count, anomaly detection, purchase filtering, solar‑farm enrichment) are hosted at https://quix.io/docs/quix-streams.

Where it runs

• Anywhere Python is available – on‑premise servers, cloud VMs, containers, or the managed Quix Cloud platform (AWS, Azure, GCP). No extra server‑side clusters beyond the Kafka brokers you already use.

Community & support

• Slack community, YouTube channel, X (Twitter) and LinkedIn for announcements and tutorials.
• Issues are tracked on GitHub; the project is Apache 2.0 licensed.

Who might use it

• Data engineers building real‑time ETL/ELT pipelines.
• Machine‑learning teams that need to stream data to/from models for inference or online learning.
• Developers of event‑driven microservices that rely on Kafka’s scalability and durability.

---

License – Apache 2.0 (see LICENSE file).