dphnAI/aphrodite-engine
Large-scale LLM inference engine
What it solves
Sonar (formerly Aphrodite Engine) is designed to optimize the serving of HuggingFace-compatible models at scale, enabling high-performance inference for multiple concurrent users.
How it works
Built on vLLM's Paged Attention technology, the engine uses continuous batching and optimized CUDA kernels to maximize throughput. It supports a wide array of quantization methods (such as AWQ, GPTQ, and GGUF) and speculative decoding techniques (like EAGLE and MTP) to speed up generation. It also provides an OpenAI-compatible API server for easy integration with various UIs.
Who it’s for
It is intended for developers and platform providers who need to serve large language models to many users simultaneously with high efficiency and low latency.
Highlights
- Broad Quantization Support: Integrates numerous formats including AQLM, AutoRound, BitNet, and Marlin.
- High-Performance Kernels: Uses PagedAttention and optimized CUDA kernels for improved inference speed.
- Advanced Sampling: Supports modern samplers like DRY, XTC, and Mirostat.
- Scalability: Features distributed and disaggregated inference, as well as multi-LoRA support.
- Multimodal Capabilities: Includes support for multimodal models.