turboderp-org/exllamav3

What it solves

ExLlamaV3 is designed to make running large language models (LLMs) on consumer-grade GPUs more accessible by reducing the memory and computational requirements through efficient quantization and optimized inference kernels.

How it works

It utilizes the EXL3 quantization format (a streamlined version of QTIP) to compress models while maintaining coherence. The library implements a Marlin-inspired GEMM kernel to achieve low latency and supports advanced inference techniques such as continuous dynamic batching, speculative decoding, and 2-8 bit cache quantization. It also provides flexible tensor-parallel and expert-parallel inference to distribute workloads across multiple consumer GPUs.

Who it’s for

It is intended for users who want to run high-performance local LLMs on modern consumer hardware, as well as developers integrating LLM inference into applications via the recommended TabbyAPI server.

Highlights

• EXL3 Quantization: A fast, efficient quantization process that can convert large models (70B+) in a few hours on a single RTX 4090.
• Broad Architecture Support: Compatible with a wide range of models including Llama, Mistral, Qwen, Deepseek, and Gemma.
• Multimodal Support: Capable of handling multimodal models such as Gemma 3/4 and Qwen 2.5 VL.
• Optimized Performance: Features speculative decoding and cache quantization to improve speed and reduce VRAM usage.