dphnAI/sonar

Large-scale LLM inference engine

What it solves

Sonar is a high-performance inference engine designed to serve Hugging Face-compatible language and multimodal models in production environments. It addresses the challenges of memory management and throughput efficiency when deploying large models at scale.

How it works

Based on vLLM, Sonar implements several advanced memory and compute optimizations. It uses continuous batching and paged KV-cache management to maximize GPU memory utilization. To further increase speed, it uses speculative decoding (via methods like EAGLE and MTP) and optimized kernels. It supports distributed serving across multiple nodes without requiring a Ray cluster, and allows for prefill/decode disaggregation through KV connectors like NIXL.

Who it’s for

It is intended for developers and ML engineers who need to deploy production-grade AI models with high throughput and low latency, supporting a wide range of hardware platforms including NVIDIA, AMD, Apple silicon, and Google TPU.

Highlights

  • High-throughput serving: Uses continuous batching and paged KV-cache management.
  • Broad hardware support: Compatible with NVIDIA CUDA, AMD ROCm, Apple silicon (Metal), and Google TPU.
  • Distributed serving: Supports tensor, pipeline, data, and expert parallelism across multiple nodes.
  • Advanced decoding: Implements speculative decoding and supports quantized weights and FP8 KV cache.
  • Multimodal support: Capable of serving image, audio, and video models. -le API Compatibility: Provides OpenAI, Anthropic, and Kobold APIs.

Related

  • Project
    sgl-project/sglang

    A high-performance serving framework for LLMs and multimodal models that provides low-latency, high-throughput inference across single GPUs and large distributed clusters.

  • Project
    waybarrios/vllm-mlx

    vllm‑mlx is an Apple‑Silicon‑only inference server that runs LLMs, vision‑language models, audio models and embeddings on Metal via the MLX framework. It offers continuous batching, paged KV‑cache, SSD‑tiered cache and other vLLM‑style performance tricks, and exposes OpenAI‑compatible and Anthropic‑compatible REST APIs so existing client libraries work out‑of‑the‑box.

  • Project
    vllm-project/vllm

    A high-throughput library for LLM inference and serving that uses PagedAttention to optimize memory management and increase efficiency.

  • Project
    Mesh-LLM/mesh-llm

    A distributed LLM serving system that pools GPUs and memory across multiple machines to run large models via an OpenAI-compatible API.

  • Project
    JetAstra/SDAR

    SDAR (Synergy of Diffusion and AutoRegression) is an open‑source LLM family that blends autoregressive training with discrete diffusion decoding. The approach yields 2‑4× faster inference while matching AR model quality, especially on science‑reasoning benchmarks. The repository provides pretrained models (1.7 B‑30 B), a simple generate script, and two high‑performance inference engines (JetEngine and LMDeploy). Training/fine‑tuning scripts use LLaMA‑Factory, and the project is MIT‑licensed and actively maintained.