jmaczan/tiny-vllm

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

tiny‑vllm – a hands‑on C++/CUDA LLM inference engine (and tutorial)

What it is – tiny‑vllm is a complete, open‑source implementation of an LLM inference server written in C++ with CUDA (and optionally HIP for AMD GPUs). It ships both the source code for the engine and a step‑by‑step educational course that walks the reader through every component of a modern transformer‑based language model, from loading weights to generating tokens.

Why it matters – Most high‑performance inference servers (e.g., vLLM, TensorRT‑LLM) are large, production‑grade codebases. tiny‑vllm deliberately strips the problem down to a single 1‑billion‑parameter Llama 3.2 model, letting learners see the exact CUDA kernels, memory layout, and batching logic that make inference fast. It therefore serves as a bridge between theory (attention, RMSNorm, FlashAttention‑style softmax, PagedAttention) and a working system you can compile and run yourself.

Key capabilities (as listed in the README)

  • Load a real LLM from a Safetensors file (tested with Llama 3.2 1B Instruct, BF16 weights).
  • Full forward pass supporting both prefill (prompt processing) and decode (single‑token generation).
  • All heavy computation performed with custom CUDA kernels, including:
    • Embedding lookup
    • RMSNorm with parallel reduction
    • Rotary positional embeddings (RoPE)
    • Grouped‑query attention (GQA)
    • Flash‑style online softmax
    • PagedAttention and a paged KV‑cache for memory‑efficient long contexts.
  • Two batching strategies:
    • Static batching – fixed‑size batch for simple workloads.
    • Continuous batching – dynamic addition/removal of requests, mimicking production servers.
  • Optional AMD GPU support via HIP/hipBLAS, reusing the same CUDA source through a thin compatibility header.

Tech stack

  • Language: C++ 17
  • GPU APIs: CUDA 13.1 (nvcc) with cuBLAS; optional HIP for AMD GPUs.
  • Model format: Safetensors (BF16 weights).
  • Dependencies: Single‑header JSON parser nlohmann/json (v3.12.0).
  • Build system: CMake (supports Ninja, can toggle -DUSE_HIP=ON).

Who should use it

  • Students / self‑learners who want to understand how LLM inference works at the kernel level.
  • Instructors looking for a teaching resource that combines code and narrative explanations.
  • Engineers curious about low‑level performance tricks (FlashAttention, KV‑cache paging) and who want a minimal, readable reference implementation.

Project status

  • The README marks all major engine components as implemented (checkboxes checked).
  • A test.sh script builds the project and runs a quick inference demo, indicating that the code compiles and produces output on the author’s Linux + RTX 5090 setup.
  • The repository is actively maintained for educational purposes; issues are encouraged for build help.

How to get started

  1. Clone the repo, adjust CUDA/GCC paths if needed.
  2. Install the CUDA Toolkit (or ROCm for AMD) and ensure a compatible GPU.
  3. Download model.safetensors for Llama 3.2 1B Instruct from Hugging Face.
  4. Run ./test.sh – the script builds the engine and runs a minimal inference test.
  5. Follow the markdown‑based course sections (e.g., Tokenization, Attention, Paged KV cache) to read the theory and inspect the corresponding source files.

Why the name – It is positioned as a “younger and smaller sibling” of the popular vLLM project, focusing on clarity and learning rather than production‑scale feature breadth.

All details above are taken directly from the repository’s README; no additional features have been inferred.

Related

  • Dispatch
    Building a High-Performance LLM Inference Engine from Scratch with tiny-vLLM

    An exploration of tiny-vLLM, a C++ and CUDA-based educational project that teaches the internals of LLM inference, from Safetensors loading to PagedAttention.

  • Project
    ModelTC/LightLLM

    LightLLM is a Python‑based, high‑performance inference and serving framework for large language models. It combines fast kernels (FlashAttention, Triton) with lightweight, token‑level KV‑cache management, supports multi‑GPU scaling, and adds research‑grade features like constrained decoding and SLA‑aware request scheduling.

  • Project
    ggml-org/llama.cpp

    A plain C/C++ implementation for high-performance LLM and VLM inference across a wide range of hardware with minimal setup.

  • Dispatch
    Fast & Efficient LLM Inference with vLLM Course Launch

    vLLM, in collaboration with DeepLearning.AI and Red Hat, has launched a free intermediate course titled Fast & Efficient LLM Inference with vLLM to teach the full AI deployment lifecycle.

  • Dispatch
    Qwen2.5-1M Release: Open‑Source 7B and 14B Models with 1M‑Token Context and vLLM‑Based Inference Framework

    Qwen releases open‑source Qwen2.5‑7B‑Instruct‑1M and Qwen2.5‑14B‑Instruct‑1M models that support up to 1 million token contexts, accompanied by an optimized vLLM‑based inference framework that delivers 3×–7× faster prefill and retains short‑task performance comparable to GPT‑4o‑mini.