theroyallab/tabbyAPI

What it solves

TabbyAPI provides a way to serve large language models (LLMs) as a web service, allowing users to generate text via an API rather than interacting with a model directly through a command-line interface or a heavy GUI. It specifically acts as the official API backend server for the ExllamaV3 backend.

How it works

Built on FastAPI, TabbyAPI wraps the ExllamaV3 backend to handle model loading and inference. It implements an OpenAI-compatible API, meaning it can be used as a drop-in replacement for OpenAI services in many applications. It supports advanced inference techniques like continuous batching with paged attention (for Nvidia Ampere GPUs and newer) and speculative decoding using draft models to increase speed.

Who it’s for

It is designed for hobbyists and developers who want to run LLMs locally or on their own hardware and expose them via a standardized API for use in other applications.

Highlights

• OpenAI Compatibility: Uses an OpenAI-compatible API for easy integration.
• Advanced Inference: Supports continuous batching, paged attention, and speculative decoding.
• Flexible Formatting: Includes support for JSON schema, Regex, EBNF, and a Jinja2 template engine for chat completions.
• Model Management: Handles loading, unloading, and downloading models directly from HuggingFace.
• Extended Capabilities: Supports embedding models and OAI-style tool/function calling.