vllm-project/tpu-inference

What it solves

It enables high-performance serving of large language models (LLMs) on Google TPU hardware. It specifically addresses the need for a unified backend that allows developers to run both PyTorch and JAX model definitions on TPUs without requiring extensive code changes, while maintaining the standardized user experience and interface of the vLLM project.

How it works

The project implements tpu-inference, a hardware plugin that creates a single lowering path for both JAX and PyTorch. This allows PyTorch models to be executed performantly on TPU hardware. It supports multiple TPU generations, including recommended versions v7x, v5e, and v6e, as well as experimental support for v3, v4, and v5p.

Who it’s for

It is designed for developers and researchers who want to deploy open-source models on Google TPUs to maximize hardware performance and reduce costs, particularly those who use PyTorch or JAX frameworks.

Highlights

• Unified Backend: Supports both PyTorch and JAX under one lowering path.
• Broad Model Support: Validated for a wide range of models including Gemma 4, Llama 3.1/3.3, and Qwen series.
• Advanced Serving Features: Includes support for chunked prefill, prefix caching, KV cache offload, and various speculative decoding methods (Eagle3, Ngram, DFlash).
• Parallelism Support: Implements Pipeline Parallelism (PP), Data Parallelism (DP), Expert Parallelism (EP), and Tensor Parallelism (TP).