pathwaycom/pathway

Pathway – Live Data Framework

What it is – Pathway is an open‑source Python library for building data pipelines that work on both batch and streaming data. Under the hood a high‑performance Rust engine (based on Differential Dataflow) runs the Python‑defined logic, giving you multithreading, multiprocessing and distributed execution while you stay in pure Python.

Why it matters for AI – The framework ships with an LLM x‑pack that bundles wrappers, parsers, embedders, splitters and an in‑memory vector index, plus integrations with LangChain and LlamaIndex. This lets you create real‑time Retrieval‑Augmented Generation (RAG) or other LLM‑driven workflows that react instantly to new data.

Key capabilities

• Unified batch‑and‑stream processing – Write a pipeline once; run it on static files or live streams without changes.
• Rich connector ecosystem – Built‑in support for Kafka, PostgreSQL, GDrive, SharePoint, Airbyte (300+ sources) and the ability to add custom Python connectors.
• Stateful operators – Joins, windowing, sorting, and arbitrary Python UDFs run on the Rust engine for speed.
• Persistence & consistency – Automatic state checkpointing; free tier gives at‑least‑once semantics, enterprise tier adds exactly‑once.
• Scalable execution – Multithreaded locally, or run containers on Docker/Kubernetes; enterprise version adds distributed cloud deployment.
• LLM helpers – Ready‑made wrappers for major LLM services, real‑time vector indexing, and compatibility with LangChain/LlamaIndex.

Typical use‑cases

• Real‑time ETL (e.g., ingesting Kafka topics, transforming, and loading to a database).
• Event‑driven alerting pipelines.
• Continuous analytics such as live regression or dashboards.
• Live LLM/RAG applications that ingest fresh documents, embed them, and answer queries instantly.

Getting started

pip install -U pathway   # Python 3.10+ on macOS or Linux

A minimal “sum of positive numbers” pipeline:

import pathway as pw

class Input(pw.Schema):
    value: int

src = pw.io.csv.read("./input/", schema=Input)
pos = src.filter(src.value >= 0)
out = pos.reduce(sum_val=pw.reducers.sum(pos.value))

pw.io.jsonlines.write(out, "output.jsonl")

pw.run()

Run the script with python my_pipeline.py or pathway spawn python my_pipeline.py to let the engine manage threads.

Deployment options

• Local – just import pathway and call pw.run().
• Docker – use the official pathwaycom/pathway image or install via pip in any Python base image.
• Kubernetes / cloud – containers scale with the enterprise edition; quick‑start on Render is documented.
• Monitoring – built‑in dashboard shows connector throughput and latency.

Performance – Benchmarks claim higher throughput than Flink, Spark, and Kafka Streams, especially for workloads that need temporal joins or iterative algorithms.

Documentation & support – Full docs at https://pathway.com/developers/, API reference, example templates, Discord community, and issue tracker.

License – Business Source License 1.1 (free for non‑commercial use and most commercial use). After four years the core code re‑licenses to Apache 2.0. Supporting repos are MIT‑licensed.

---

Bottom line – Pathway lets data engineers and AI developers write Python pipelines that run at “Rust speed”, handling both historic and live data, with first‑class support for LLM‑centric RAG workflows.