icey1287/SuperMew

SuperMew – An Auditable, RAG‑first Agent Platform

What it is – SuperMew is a self‑hosted AI‑agent framework that treats every user interaction as a durable, replayable workflow rather than a one‑off HTTP call. It stitches together conversation threads, runs, events, and check‑points so that a chat can be paused for human‑in‑the‑loop (HITL) review, resumed later, and fully audited.

Core ideas

Persisted domain objects – Thread, Message, Run, Event, Checkpoint, DocumentVersion are stored in PostgreSQL (metadata) and Milvus (vector chunks). This lets you replay or debug any past interaction.
Two‑stage document publishing – New document versions are built in an isolated candidate scope, verified, then atomically swapped in via a PostgreSQL CAS operation. Ongoing queries never see a half‑built index.
Hybrid RAG pipeline – Dense vector search (Milvus) + native BM25 are fused with Reciprocal Rank Fusion, optionally re‑ranked, judged for evidence quality, and can fall back to a single HyDE or step‑back rewrite. All intermediate steps are recorded in a rag_trace that the UI visualises.
Skill / Tool registry – Fixed‑version skills (knowledge‑base lookup, weather, read‑only SQL, web‑research, sandboxed code execution, restricted HTTPS JSON) are declared in a registry. Each tool’s schema is disclosed to the agent only after the run’s permission check, and guardrails enforce allow/deny/approval policies.
Model control plane – Admins define Model Profiles (no secrets) and assign them to four roles: Answer, Fast, Grader, Evaluator. When a Run or Evaluation Job is created the exact model snapshot is frozen, guaranteeing reproducibility.
RAG evaluation framework – Versioned datasets, baseline comparisons, CI‑gate style quality checks, and a persistent evaluation worker let you measure correctness, groundedness, relevance, completeness, unsupported claims, and conflict disclosure automatically.
Security & auth – Short‑lived in‑memory access tokens, rotating opaque refresh tokens stored in HttpOnly cookies, RBAC, per‑endpoint rate‑limit HMAC, CSP/Headers, and a read‑only SQL sandbox with AST and RLS checks.

Why it matters – By persisting every step, SuperMew makes it possible to:

Audit exactly which documents, tools, and model outputs produced a response.
Resume a paused conversation after a human reviewer adds clarification.
Version‑control knowledge bases and model configurations without breaking existing runs.
Benchmark RAG pipelines end‑to‑end in a production‑like environment.

Architecture at a glance

Component	Tech	Role
API server	FastAPI (Python 3.12)	HTTP, SSE, auth, thread/run orchestration, static front‑end serving
Index worker	Python module `backend.workers.indexing`	Build immutable document versions, write Milvus vectors, manage candidate scope
RAG evaluation worker	Python module `backend.workers.evaluation`	Execute offline evaluation jobs, compute metrics, store results
Vector store	Milvus (dense + native BM25)	Fast nearest‑neighbor retrieval
Metadata store	PostgreSQL + SQLAlchemy + Alembic	Threads, runs, events, model snapshots, document catalogs
Cache / low‑latency notifications	Redis	Event push, rate‑limit counters
Object storage	MinIO (used by workers)	Uploaded raw documents
Frontend	Vite + Vue 3 + TypeScript + Pinia	UI for chat, skill centre, admin panels, evaluation workbench

Getting started (local development)

Prerequisites – Python 3.12+, uv package manager, Node 20+, Docker Compose.
Configure – Copy .env.example to .env and fill in:
- Model identifiers (MODEL, FAST_MODEL, …)
- JWT_SECRET_KEY (≥32 random chars)
- Optional admin invite code.
Start dependencies – docker compose up -d (Postgres, Redis, etcd, MinIO, Milvus, Attu).
Install Python deps – uv sync --frozen.
Build frontend – cd frontend && npm ci && npm run build && cd ...

Run migrations & sanity checks –

uv run --frozen alembic upgrade head
uv run --frozen python -m backend.tools.registry_cli validate

Launch the three processes – ./scripts/start.sh (API, index worker, evaluation worker). Use --no-reload to disable hot‑reload.
Open http://127.0.0.1:8000/ for the UI, http://127.0.0.1:8000/docs for OpenAPI, and http://127.0.0.1:8080/ for Milvus Attu.

Production checklist (summary)

Deploy the same three services (API, index worker, evaluation worker) under a supervisor (systemd, k8s, etc.) ensuring they share the same code version and UPLOAD_DIR.
Set APP_ENV=production and provide real secrets for JWT, rate‑limit HMAC, DB passwords, and model provider keys.
Enable secure cookies (AUTH_REFRESH_COOKIE_SECURE=true).
Start workers before the API, run a health‑check, then perform a minimal end‑to‑end test (create a thread, upload a doc, run a RAG query, start a tiny evaluation job) before opening the public endpoint.
Periodically run the cleanup task python -m backend.auth.cleanup to prune expired refresh‑token ledgers.

Typical use cases

Use case	How SuperMew helps
Enterprise knowledge‑base Q&A	Upload PDFs/Docs, version them, and let agents retrieve with hybrid search while preserving audit trails.
Human‑in‑the‑loop support	Pause a run at a `Checkpoint`, let a reviewer edit or approve a tool call, then resume without losing context.
Regulated environments	Read‑only SQL assistant with strict allow‑lists, sandboxed code execution, and policy‑driven tool guardrails satisfy compliance needs.
Model / RAG benchmarking	Define a dataset, run the evaluation worker, compare against baselines, and gate releases based on quality metrics.
Custom skill integration	Add a new HTTP‑JSON tool or a domain‑specific skill via the registry; the platform automatically handles versioning, permission checks, and UI exposure.

TL;DR

SuperMew is a full‑stack, open‑source platform for building production‑grade AI agents that rely on Retrieval‑Augmented Generation. It emphasizes durability, auditability, human‑in‑the‑loop control, and secure, versioned tooling. If you need a self‑hosted RAG system where every step can be inspected, replayed, or gated, SuperMew provides the complete stack—from PostgreSQL‑backed event sourcing to a Vue‑based UI and a robust evaluation framework.