farshed/sage

Sage – Talk to LLMs with your voice

What it is – Sage is an open‑source application that lets you have a spoken conversation with a large language model (LLM). It stitches together a speech‑to‑text model, an LLM (self‑hosted via Ollama or accessed through a cloud API), and a text‑to‑speech component, all running locally so your data never leaves the machine.

Why it matters – Most voice‑assistant demos rely on proprietary services. Sage shows that a full voice‑LLM pipeline can be assembled from freely available models and run on consumer hardware with modest latency.

Core pieces

Piece	Role	Example model used
Speech‑to‑text	Turns your microphone input into text	Whisper (ggml‑large‑v3‑turbo)
LLM	Generates the response	Any model served by Ollama (e.g., `deepseek-r1:7b`) or cloud providers (OpenAI, Anthropic, Deepseek, Together.ai)
Text‑to‑speech	Reads the LLM reply aloud	Not explicitly named, but supplied via the `voices.json` catalog

All three stages are wired together in a small Node‑style server written in Bun (a fast JavaScript runtime). The architecture diagram in the README shows the data flow: microphone → Whisper → LLM → voice synthesis → speakers.

Getting started

Choose a language model
- Self‑hosted – Install Ollama, pick a model, and set OLLAMA_MODEL in a .env file.
- Cloud – Add the appropriate *_API_KEY and *_MODEL variables for OpenAI, Anthropic, Deepseek, or Together.ai.
Install the stack
- Docker route (simpler) – Place three model files in a models/ folder, run bun docker-build then bun docker-run. UI appears at http://localhost:3000 (note: Whisper runs ~4‑5× slower inside Docker).
- Native route (faster) – Install Bun, Rust, OpenSSL, LLVM/Clang, and CMake. Run the platform‑specific setup‑unix.sh or setup‑win.bat to download weights and compile the Whisper binaries. Start with bun start.
Talk – Open the web UI, speak into your mic, and hear the model’s answer.

Design goals highlighted in the README

Open – Uses community‑maintained speech models (Whisper, Kokoro voice encoder).
Efficient – Targets consumer‑grade CPUs/GPUs; the only heavy part is Whisper, which can run in real‑time on modern laptops.
Self‑hosted – No external services required unless you opt for a cloud LLM.
Modular – Switching LLM providers is just an environment‑variable change.

Roadmap (as of the README)

✅ Docker packaging (already available)
⬜ CUDA acceleration for faster inference
⬜ Custom Ollama endpoint support
⬜ Multilingual speech support
⬜ Expose Whisper configuration options
⬜ System‑prompt customization
⬜ General pipeline optimisations
⬜ Publish Sage as a reusable library

Bottom line

Sage is a practical, end‑to‑end demo of a voice‑first AI assistant built entirely from open‑source components. It is useful for hobbyists who want to experiment with spoken LLM interactions without relying on closed APIs, and it provides a clear example of how to glue speech models, LLM back‑ends, and TTS together in a lightweight, self‑contained package.