deepseek-ai/DeepInfer

DeepInfer – High‑performance LLM inference engine

What it is – DeepInfer is an open‑source inference framework built by DeepSeek AI to serve their large language models (currently the DeepSeek‑V4.1‑Flash family) at production scale. It focuses on offline, high‑throughput generation on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

Key capabilities

• Optimized kernels – integrates FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang ops for fast attention, matrix multiplication and sampling.
• Parallelism – supports attention data‑parallelism, balanced compute‑parallelism, and mixture‑of‑experts (MoE) expert‑parallelism with load‑balancing.
• Draft‑token verification (DSpark) – generates speculative tokens, then uses confidence‑based verification to reduce decode latency.
• SWA bounded replay – re‑uses a limited suffix of the KV cache to cut pre‑fill work.
• Asynchronous batch scheduler – overlaps CPU scheduling with GPU execution, giving “zero‑overhead” batching.
• Batch‑invariant & deterministic sampling – guarantees identical outputs regardless of batch composition when enabled (NVIDIA only).
• Structured outputs – can constrain generation to a regex, JSON schema or EBNF grammar via SamplingParams.

Supported hardware & models

• NVIDIA SM100 GPUs (CUDA 13)
• Huawei Ascend 950 NPUs (CANN 9.2)
• Currently ships with the checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017.

How to get started

1. Install Linux, GCC ≥ 13, Python 3.12, and the appropriate accelerator stack (CUDA or CANN).
2. Clone the repo, create a fresh Python 3.12 virtual environment, set DEEPINFER_ACCELERATOR=nvidia (or ascend), and run ./build.sh.
3. Run the provided example:

   python examples/offline_inference.py
   

   The first run will pull the model from Hugging Face; you can point to a local copy with DEEPINFER_HF_MODEL_DIR.

Performance – Benchmarks (run on 4 SM100 GPUs for pre‑fill and 64 for decode) show trade‑offs between throughput and latency; the repo includes scripts to reproduce the Pareto curves.

Evaluation – An eval/ folder contains scripts and instructions for measuring accuracy, latency and other metrics.

License – Apache 2.0.

Relation to other projects – DeepInfer builds on ideas from vLLM, SGLang and NVIDIA TensorRT‑LLM, adding DeepSeek‑specific kernels and scheduling optimizations.

---

All details are taken directly from the repository’s README.