dphnAI/sonar
Large-scale LLM inference engine
What it solves
Sonar is a high-performance inference engine designed to serve HuggingFace-compatible models at scale. It addresses the challenge of efficiently managing GPU memory and handling multiple concurrent users without sacrificing speed.
How it works
Built on vLLM's Paged Attention technology, Sonar optimizes model serving by utilizing continuous batching and optimized CUDA kernels. It supports a wide array of quantization methods (such as AWQ, GPTQ, and GGUF) to reduce memory footprints and employs speculative decoding techniques (like EAGLE and MTP) to accelerate generation. It provides an OpenAI-compatible API server for easy integration with various UIs.
Who it’s for
It is intended for developers and platform providers who need to deploy LLMs for large numbers of users, as well as those integrating AI chat platforms and API infrastructure.
Highlights
- PagedAttention: Efficient K/V cache management to maximize throughput.
- Extensive Quantization Support: Compatible with numerous formats including AQLM, AutoRound, and BitNet.
- Advanced Sampling: Support for modern samplers like DRY, XTC, and Mirostat.
- Scalability: Features distributed and disaggregated inference and multi-LoRA support.
- Multimodal Support: Ability to handle multimodal models.