patchy631/time-to-first-token

A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking.

What it solves

This project provides a structured, 10-week learning roadmap for engineers to move from theoretical knowledge of Large Language Model (LLM) inference to deploying and optimizing a production-ready inference service. It solves the problem of fragmented learning by focusing on a single, growing artifact: an OpenAI-compatible service that is instrumented, load-tested, and optimized for cost and performance.

How it works

The roadmap is organized into 50 sessions (30 minutes each) that guide the user through a specific sequence of technical milestones:

  1. Mental Models: Understanding the roofline model, arithmetic intensity, and the difference between compute-bound prefill and memory-bandwidth-bound decode.
  2. Deployment: Setting up vLLM and SGLang to serve models (e.g., Llama-3.1-8B) and exploring their internals like PagedAttention and RadixAttention.
  3. Observability: Building a measurement stack using Prometheus and Grafana to track metrics like Time to First Token (TTFT) and throughput.
  4. Performance Tuning: Implementing and benchmarking quantization (FP8, INT4/AWQ), speculative decoding, and KV cache eviction.
  5. Infrastructure & Economics: Deploying to Kubernetes with custom autoscaling and building a cost-aware router to manage unit economics.

Who it’s for

Engineers who are comfortable with Python, the command line, and transformer architectures, but who may lack experience in CUDA, Kubernetes, or professional LLM serving.

Highlights

  • Practical Focus: Every session feeds into a single production-grade serving stack.
  • Measurement-First: Emphasizes instrumentation and load testing (up to 1000+ concurrent requests) before applying optimizations.
  • Hardware-Aware: Teaches the roofline model to explain why specific optimizations like quantization or continuous batching work.
  • Comprehensive Stack: Covers everything from raw GPU rentals to Kubernetes Helm charts and cost-routing middleware.

Related

  • Project
    llm-d/llm-d

    llm‑d is an open‑source, CNCF‑sandbox project that adds production‑grade routing, KV‑cache tiering, and expert‑parallelism to existing LLM model servers (vLLM, SGLang). Delivered as Helm charts for Kubernetes, it lets organizations serve large language models at scale with lower latency and higher throughput across many accelerator types.

  • Dispatch
    Building a High-Performance LLM Inference Engine from Scratch with tiny-vLLM

    An exploration of tiny-vLLM, a C++ and CUDA-based educational project that teaches the internals of LLM inference, from Safetensors loading to PagedAttention.

  • Dispatch
    Fast & Efficient LLM Inference with vLLM Course Launch

    vLLM, in collaboration with DeepLearning.AI and Red Hat, has launched a free intermediate course titled Fast & Efficient LLM Inference with vLLM to teach the full AI deployment lifecycle.

  • Project
    algorithmicsuperintelligence/optillm

    An OpenAI API-compatible inference proxy that implements 20+ optimization techniques to improve LLM reasoning accuracy without requiring model training.

  • Project
    ulab-uiuc/LLMRouter

    An open-source library for intelligent LLM routing that dynamically selects the most suitable model for each query to optimize inference cost and performance.