pathwaycom/serviette

What it is

Pathway Live Data Framework AI Pipelines is a collection of ready‑to‑run application templates that let you build retrieval‑augmented generation (RAG) services on top of live data sources (files, Google Drive, SharePoint, S3, Kafka, PostgreSQL, etc.). Each template bundles the code for data ingestion, vector/hybrid indexing, LLM calling and an HTTP API (optionally a Streamlit UI). The heavy lifting is done by the open‑source Pathway library, which synchronises data in real time and provides an in‑memory vector store (via usearch) and full‑text index (via Tantivy).

Why it matters

• Live updates – documents added, removed or changed are re‑indexed automatically, so the RAG system always answers from the freshest knowledge.
• All‑in‑one stack – no external vector DB, cache or API framework to provision; Pathway supplies a built‑in vector index and FastAPI server.
• Scalable templates – examples claim to handle millions of pages and support multimodal inputs (PDF tables, slides, video captions) while keeping latency low.
• Flexibility – you can swap data sources, change the index type, or replace the LLM with a single line change.

Who should use it

• Product teams that need a quick prototype of an enterprise search or document‑question‑answering service without wiring together separate components.
• Data engineers who want a reproducible pipeline for continuously ingesting corporate data into a searchable vector store.
• Developers looking for a reference implementation of RAG patterns (adaptive RAG, multimodal RAG, unstructured‑to‑SQL) that can be run locally or deployed to cloud/on‑prem.

How to get started

1. Clone the repo and pick a template folder under templates/ (e.g., question_answering_rag).
2. Follow the template’s README.md – usually it’s a one‑liner to build a Docker image and run it, optionally launching a Streamlit UI.
3. Configure the data source (local folder, Google Drive, S3, etc.) and the LLM you want to use (OpenAI GPT‑3/4, Mistral via Ollama, etc.) via environment variables.
4. The container starts a FastAPI server exposing /query (or similar) and, if included, a Streamlit front‑end for interactive testing.
5. For production, push the Docker image to your cloud registry and run it on GCP, AWS, Azure, Render, or on‑premises.

What’s inside

Template	Core use‑case	Highlights
Question‑Answering RAG	Simple doc‑QA over PDFs/DOCX	Live sync, any LLM, demo REST endpoint
Live Document Indexing	Vector‑store‑as‑a‑service	Can be used as a backend for LangChain or LlamaIndex
Multimodal RAG (GPT‑4o)	Extract tables/charts from PDFs, index them	Uses GPT‑4o for parsing, handles charts & tables
Unstructured‑to‑SQL	Convert financial PDFs to a SQL table and answer via generated queries
Adaptive RAG	Reduces token usage up to 4× while keeping accuracy
Private RAG (Mistral + Ollama)	Fully local stack, no external API calls
Slides AI Search	Index PowerPoint & PDF slides, multimodal retrieval
Video RAG (TwelveLabs)	Turn video into rich text, embed with Marengo, query video content

Community & support

• Discord channel for help (discord.gg/pathway).
• Issues are filed on the main Pathway repo (github.com/pathwaycom/pathway).
• Official docs and more templates are listed on the Pathway website.

---

TL;DR – This repo provides Docker‑ready RAG application templates built on the Pathway Live Data Framework, enabling real‑time indexing of many data sources and serving answers via LLMs, with no external vector‑DB or cache required.