turboderp-org/exllamav3

An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs

What is ExLlamaV3?

ExLlamaV3 is a Python library that lets you run large language models (LLMs) locally on a consumer‑grade GPU. It focuses on fast inference by providing:

EXL3 quantization – a low‑bit (2‑8 bit) weight format derived from the QTIP research paper, plus optional low‑bit cache quantization, which dramatically reduces memory use while keeping quality high.
Parallel inference – tensor‑parallel and expert‑parallel execution so you can split a model across multiple GPUs or across CPU and GPU when GPU memory is limited.
CPU off‑loading – MoE (Mixture‑of‑Experts) models can run on a modest GPU by moving parts of the computation to the CPU (AVX2/AVX512 supported).
Dynamic generation features – continuous batching, speculative decoding, and multimodal (image‑text) support.
Integrations – a Hugging Face model loader, a small Transformers‑compatible wrapper, and an OpenAI‑style API via the separate TabbyAPI server.

In short, ExLlamaV3 is an inference‑engine that makes it practical to run state‑of‑the‑art LLMs (Llama 3, Gemma 4, Qwen 3.5, etc.) on a single RTX 40xx/50xx card or a modest multi‑GPU rig.

Key Features (jargon‑light)

Feature	What it means for you
EXL3 quantization	Model weights are stored in 2‑8 bit formats, cutting memory needs by up to 4‑8× compared with full‑precision models.
Parallel & expert‑parallel inference	The library can split the work across several GPUs or between GPU and CPU, letting you run larger models than would fit on one device.
CPU off‑loading	Even MoE models that normally need many GPUs can be run with a single GPU plus CPU resources.
Dynamic batching & speculative decoding	The engine keeps the GPU busy by grouping many requests together and guessing next tokens, which speeds up response time.
Multimodal support	Some models that accept images (e.g., Qwen‑2.5 VL) are supported out of the box.
Easy integration	Use the provided `examples/chat.py` CLI, plug into Hugging Face Transformers, or expose the model through an OpenAI‑compatible HTTP API (TabbyAPI).
Broad model coverage	Over 40 model families are listed, from Llama 1‑3 to Gemma 4, Qwen 3‑5, Mistral, Mixtral, and many MoE variants.

Getting Started (Installation)

Make sure you have a recent PyTorch build (CUDA 12.4+). The library does not install PyTorch for you.
Choose one of the following:
- Pre‑built wheel – download the matching exllamav3‑<version>+cuXXX.torch2.x.x‑...whl from the releases page and pip install <url>.
- PyPI – pip install exllamav3 (you’ll need a CUDA toolkit and a C++ compiler to build the native extension).
- Build from source – clone the repo and use uv or pip with the appropriate --extra cuXXX flag that matches your CUDA version (e.g., cu128 for CUDA 12.8). The README gives four detailed options, including how to pin a specific PyTorch version.
After installation, run the example CLI:
```
python examples/chat.py -m <model_folder> -mode <prompt_mode>
```
Use -h to see all the many options.

Typical Workflow

Download a model from Hugging Face (or use an already‑quantized EXL3 model from the community collection).
Convert the model to EXL3 format:
```
python convert.py -i <hf_dir> -o <exl3_dir> -b 4   # 4‑bit example
```
The converter handles Hessian‑based calibration automatically; larger models may take a few hours on a high‑end GPU.
Run inference via the CLI, the Transformers wrapper, or start a TabbyAPI server to expose an OpenAI‑compatible endpoint.
(Optional) Fine‑tune or calibrate further using the optimize.md guide.

Who Might Use This?

Researchers & hobbyists who want to experiment with the latest LLMs without renting cloud GPUs.
Developers building local AI assistants that need fast response times and low memory footprints.
Teams deploying private inference services – the TabbyAPI server lets you serve models behind your firewall with an OpenAI‑style API.
Anyone interested in quantization research – the library implements the QTIP‑based EXL3 method and provides tools to inspect/adjust the process.

Community & Resources

Discord – https://discord.gg/NSFwVuCjRq for help, model sharing, and development discussion.
Hugging Face model hub – a curated collection of ready‑to‑use EXL3‑quantized models.
Acknowledged projects – builds on PyTorch, FlashAttention, QTIP, Marlin, and other open‑source kernels.

TL;DR

ExLlamaV3 is a high‑performance, quant‑aware inference engine for running modern LLMs on consumer GPUs. It offers low‑bit weight storage, flexible parallelism, CPU off‑loading, and easy integration with existing Python tooling or an OpenAI‑compatible API.