ServiceStack/llms

ServiceStack/llms – Quick Overview

What it is – llms.py is a lightweight command‑line and web‑API tool that lets you talk to a variety of large language models (LLMs) without needing an internet connection. It aims to be a self‑hosted, privacy‑first alternative to the Open WebUI project.

Key capabilities

• Offline multi‑LLM access – Hook up any locally‑hosted LLM (e.g., Llama‑2, Mistral, etc.) and query it from the same interface.
• CLI & HTTP API – Use a simple terminal command or send REST calls to get completions, making it easy to script or embed in other apps.
• Chat‑style UI – A browser‑based front‑end mimics ChatGPT‑like conversations, storing chat history only in the browser’s local storage so nothing leaves the user’s machine.
• MCP client integration – Optional extension that connects conversations/agents to remote “MCP” tools (OAuth‑protected per‑user endpoints). Approvals are editable and the system can recover from failures without replaying uncertain calls.
• Sharing extensions – share_static and share_llmspy let you expose static assets or whole conversation sessions to others, controlled via configuration files.

How it works

1. Run the Python script (llms.py) – it starts a tiny web server exposing a REST API and serves the bundled UI.
2. Configure LLM back‑ends – point the server at locally‑running model services (via HTTP, gRPC, etc.).
3. Interact – open the UI in a browser, type prompts, and the server forwards them to the selected model, returning the response.
4. Optional MCP layer – if you enable the extensions/mcp_client module, the server can forward certain calls to remote tools that require OAuth authentication, with per‑user approval stored on the server.

Who might use it

• Developers who want a quick, self‑hosted ChatGPT‑style UI for experimenting with local LLMs.
• Teams that need to keep prompt data private and prefer browser‑local storage over cloud databases.
• Researchers building agents that need to call external services (via the MCP client) while keeping the core conversation offline.

Where to learn more

• Project website: https://llmspy.org
• Repository: https://github.com/ServiceStack/llmspy.org (the README you’re reading)
• Extension docs: llms/extensions/mcp_client/README.md and docs/SHARING.md

---

All details above are taken directly from the repository’s README; no additional features have been inferred.