theroyallab/tabbyAPI

TabbyAPI – Fast, Open‑source LLM serving with an OpenAI‑compatible REST API

What it is

• A Python 3.10‑3.14 FastAPI application that wraps the ExllamaV3 inference engine. It turns locally‑stored LLM checkpoints into a web service that speaks the same JSON schema as OpenAI’s v1/completions and v1/chat/completions endpoints.

Why it matters

• ExllamaV3 is known for its paged‑attention, continuous‑batching and GPU‑accelerated inference on Nvidia Ampere+ cards. TabbyAPI exposes those performance gains behind a familiar HTTP API, making it easy to plug into existing tooling (e.g., LangChain, AI‑Horde, custom front‑ends) without needing to write CUDA code yourself.

Key features (as listed in the README)

• OpenAI‑compatible API – drop‑in replacement for chat/completions and completions endpoints.
• Model lifecycle – load, unload, and switch models at runtime.
• HuggingFace downloader – fetch models directly from the Hub.
• Embedding model support – serve vector embeddings alongside text generation.
• Schema‑aware prompting – JSON schema, regex, and EBNF validation for structured outputs.
• AI‑Horde integration – optional participation in the distributed inference network.
• Speculative decoding – use a fast “draft” model to accelerate generation.
• Proxy layer – override client‑side parameters or samplers on the fly.
• Jinja2 templating – flexible prompt construction that mirrors HuggingFace chat format.
• Async concurrency – multiple requests handled simultaneously via asyncio.
• Tool/function calling – OAI‑style function calling support.
• Embeddings stack (optional) – extra Docker tag includes the infinity‑emb stack for high‑throughput vectorisation.

Supported model types

• Exl3 (the ExllamaV3 format) – recommended for best speed.
• FP16 / BF16 checkpoints.
• Works on Nvidia Ampere or newer GPUs with paged‑attention batching.

How to get started

1. Docker (recommended) – pull the pre‑built image:

   docker pull ghcr.io/theroyallab/tabbyapi:latest   # CUDA 12.8
   docker run --gpus all -p 5000:5000 \
       -v /path/to/models:/app/models \
       ghcr.io/theroyallab/tabbyapi:latest
   

   The API will be reachable at http://localhost:5000.
2. Alternative tags – cu13 for CUDA 13, latest-extras for the embeddings stack.
3. From source – clone the repo, install the Python dependencies, and run the FastAPI app (uvicorn tabbyapi.main:app). See the Wiki for detailed steps and Docker‑Compose examples.
4. Configuration – model directory, port, and optional settings are controlled via environment variables or a config.yaml file (documented in the Wiki).

Typical use cases

• Personal or hobbyist LLM serving on a desktop workstation.
• Rapid prototyping of ChatGPT‑style applications without paying for cloud APIs.
• Integration with self‑hosted UI projects like SillyTavern or Text Generation WebUI.
• Experimentation with speculative decoding, function calling, or custom prompt templates.

Limitations

• Described as a hobby project, not intended for production‑grade scaling or high‑availability deployments.
• Rolling‑release status means breaking changes may occur; you may need to reinstall dependencies after updates.

License

• AGPL‑v3 – any modifications that are deployed as a service must also be made publicly available under the same license.

Community & support

• Official Discord server (link in README) for troubleshooting and feature requests.
• Contributions are welcomed via pull requests; the repo provides issue/PR templates.

Acknowledged upstream projects

• ExllamaV2/V3, Aphrodite Engine, infinity‑emb, FastAPI, Text Generation WebUI, SillyTavern, among others.

---

Bottom line: TabbyAPI is a lightweight, open‑source gateway that lets you run modern LLMs locally with an OpenAI‑style HTTP API, leveraging the fast ExllamaV3 backend. It’s best suited for developers and hobbyists who want a self‑hosted alternative to commercial LLM APIs.