GeeeekExplorer/nano-vllm

Nano vLLM

What it solves

Nano-vLLM provides a lightweight, readable alternative to the full vLLM implementation, allowing users to achieve high-speed offline inference without the complexity of a massive codebase.

How it works

It is a from-scratch implementation of a vLLM-style inference engine written in approximately 1,200 lines of Python. It achieves performance comparable to vLLM by utilizing an optimization suite that includes prefix caching, Tensor Parallelism, Torch compilation, and CUDA graphs.

Who it’s for

Developers and researchers who need a fast offline inference engine but prefer a clean, readable codebase for easier understanding or customization.

Highlights

  • High Performance: Delivers offline inference speeds comparable to vLLM.
  • Minimalist Code: Implemented in roughly 1,200 lines of Python.
  • Advanced Optimizations: Supports CUDA graphs, Torch compilation, Tensor Parallelism, and prefix caching.
  • vLLM-like API: Mirrors the vLLM interface for ease of use.

Related

  • Project
    jmaczan/tiny-vllm

    tiny‑vllm is an open‑source C++/CUDA LLM inference engine (with optional AMD‑HIP support) that loads a Llama 3.2 1B model from Safetensors and implements the full forward pass—including custom kernels for embeddings, RMSNorm, RoPE, grouped‑query attention, Flash‑style softmax, and paged KV‑cache. It doubles as a step‑by‑step tutorial covering every component of transformer inference, aimed at learners and educators.

  • Project
    ModelTC/LightLLM

    LightLLM is a Python‑based, high‑performance inference and serving framework for large language models. It combines fast kernels (FlashAttention, Triton) with lightweight, token‑level KV‑cache management, supports multi‑GPU scaling, and adds research‑grade features like constrained decoding and SLA‑aware request scheduling.

  • Project
    mudler/vllm.cpp

    A lightweight C++20 inference engine that mirrors vLLM's features and throughput but removes the Python/PyTorch dependency for a 140x smaller installation footprint.

  • Project
    EricLBuehler/candle-vllm

    A high-performance inference and serving platform for local LLMs featuring an OpenAI-compatible API, multi-GPU/multi-node support, and aggressive KV cache compression.

  • Dispatch
    Building a High-Performance LLM Inference Engine from Scratch with tiny-vLLM

    An exploration of tiny-vLLM, a C++ and CUDA-based educational project that teaches the internals of LLM inference, from Safetensors loading to PagedAttention.