jmaczan/tiny-vllm

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

What it solves

Deploying large language models requires an inference server that can run the model efficiently on GPUs, handling multiple requests concurrently. tiny-vllm addresses this by providing a from-scratch inference engine written in C++ and CUDA, demonstrating how to load a real model (Llama 3.2 1B Instruct) and execute the full forward pass with performance-oriented techniques. It also serves as a learning resource, teaching the underlying math and engineering decisions.

How it works

The engine reads a Safetensors model file, extracts weights in bfloat16, and implements the Llama 3.2 architecture. It includes CUDA kernels for embeddings, RMSNorm, RoPE, attention (with FlashAttention-like online softmax), and feed-forward layers. It uses KV caching, static and continuous batching, and PagedAttention to efficiently manage memory and concurrent requests. The code builds with CMake and runs on NVIDIA GPUs via CUDA/cuBLAS, with optional AMD support via ROCm/HIP.

Who it’s for

Developers interested in understanding LLM inference internals, students learning CUDA and high-performance computing, and lecturers who want a hands-on teaching resource. It assumes a basic grasp of C++ and some familiarity with neural networks, but walks through each concept.

Highlights

  • Full implementation of an inference server with prefill and decode steps
  • CUDA kernels for all major operations, with optional HIP support for AMD GPUs
  • Includes PagedAttention and continuous batching for throughput
  • Educational: each component is explained in detail, derived from first principles
  • Minimal external dependencies (only nlohmann/json)

Related

  • Dispatch
    Building a High-Performance LLM Inference Engine from Scratch with tiny-vLLM

    An exploration of tiny-vLLM, a C++ and CUDA-based educational project that teaches the internals of LLM inference, from Safetensors loading to PagedAttention.

  • Project
    ModelTC/LightLLM

    LightLLM is a Python‑based, high‑performance inference and serving framework for large language models. It combines fast kernels (FlashAttention, Triton) with lightweight, token‑level KV‑cache management, supports multi‑GPU scaling, and adds research‑grade features like constrained decoding and SLA‑aware request scheduling.

  • Project
    ggml-org/llama.cpp

    A plain C/C++ implementation for LLM and VLM inference with minimal setup and state-of-the-art performance across a wide range of hardware.

  • Project
    vllm-project/vllm

    vLLM is an open‑source library for high‑throughput, low‑latency serving of large language models. It introduces PagedAttention for efficient KV‑cache handling, supports many quantisation formats, integrates fast attention kernels, and offers an OpenAI‑compatible API that runs on GPUs, CPUs and other accelerators.

  • Project
    1CatAI/1Cat-vLLM

    1Cat‑LLM is a vLLM fork that optimises LLM inference for NVIDIA Volta (Tesla V100) GPUs, offering custom kernels, FP8/E4M3 KV‑cache, GGUF support and speculative decoding to achieve up to ~260 tokens / s on 27‑B models while reporting quality‑aware benchmarks.