jmaczan/tiny-vllm

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

What it solves

tiny-vllm is a high-performance LLM inference engine designed to be a smaller, educational version of vLLM. It provides a practical implementation of the complex operations required to run Large Language Models (LLMs) on NVIDIA GPUs, allowing users to learn how to build an inference server from scratch using C++ and CUDA.

How it works

The project implements a full LLM forward pass (prefill and decode) using CUDA kernels for all computations. It specifically targets the Llama 3.2 1B Instruct architecture and loads model weights from Safetensors files. Key technical components include:

  • Memory Management: Implements a KV cache and PagedAttention to optimize memory usage during token generation.
  • Batching: Supports both static and continuous batching to handle multiple requests in parallel.
  • Computational Optimizations: Uses online softmax and FlashAttention-like mechanisms to increase efficiency.
  • Core Operations: Implements essential LLM components such as RMSNorm, RoPE, SiLU activation, and Feed Forward Networks (FFN) directly in CUDA.

Who it’s for

This project is for developers, students, and lecturers who want to understand the internal mechanics of LLM inference engines. It is particularly useful for those with a background in C++ and CUDA who want to a hands-on way to learn the mathematics and engineering behind modern LLM serving.

Highlights

  • Full CUDA Implementation: All computations are performed with custom CUDA kernels for maximum hardware efficiency.
  • Advanced Inference Techniques: Includes PagedAttention and continuous batching, mirroring features found in production-grade engines like vLLM.
  • Educational Focus: Structured as both a source code repository and a guided course that derives the necessary math and logic from scratch.
  • Llama 3.2 Support: Specifically optimized to run the Llama 3.2 1B Instruct model.