Wenyueh/MinivLLM

What it solves

It provides a custom, self-contained implementation of a vLLM-style inference engine. It is designed to demonstrate and benchmark the efficiency of different attention mechanisms—specifically Flash Attention for prefilling and Paged Attention for decoding—compared to standard PyTorch implementations.

How it works

The project implements a complete LLM inference pipeline including a scheduler, block management for KV cache, and a runner for executing prefilling and decoding phases. It uses a custom engine to process batched prompts through a model (such as a small version of Qwen3). To optimize memory and speed, it incorporates a Triton-based Flash Attention kernel for the prefilling phase and a specialized Triton kernel for Paged Attention during the decoding phase.

Who it’s for

Developers and researchers interested in the internal mechanics of LLM inference engines, specifically those wanting to understand how memory-efficient attention and KV cache management are implemented from scratch.

Highlights

• Self-contained Attention: Includes its own implementations of Paged Attention and Flash Attention.
• Performance Benchmarking: Provides dedicated scripts to compare prefilling (Flash Attention vs. PyTorch) and decoding (Triton kernels vs. PyTorch).
• Full Pipeline: Implements the entire engine logic, including sequence definition, block management, and iteration-based scheduling.
• Multi-GPU Support: Supports multi-GPU configurations by adjusting the world size in the configuration.