tile-ai/TileRT
Tile-Based Runtime for Ultra-Low-Latency LLM Inference
What it solves
TileRT is designed for ultra-low-latency LLM inference, specifically targeting scenarios where the time per output token (TPOT) must be minimized. It addresses the bottleneck of traditional inference systems that prioritize high-throughput batch processing over the responsiveness required for real-time applications like high-frequency trading, interactive AI, and AI-assisted coding.
How it works
TileRT uses a compiler-driven, tile-level runtime engine. It decomposes LLM operators into fine-grained tile-level tasks and dynamically reschedules computation, I/O, and communication across multiple devices in a highly overlapped manner to minimize hardware idle time and maximize utilization.
Who it’s for
It is intended for developers and organizations requiring millisecond-level responsiveness from massive models (hundreds of billions of parameters) on high-end hardware, such as NVIDIA B200 GPUs.
Highlights
- Extreme Performance: Capable of breaking 1000 tokens/s on a 1-trillion-parameter model on a single 8-GPU node.
- Multi-Token Prediction (MTP): Supports generating multiple tokens per forward pass to reduce sequential decoding depth.
- Model Support: Provides specialized backends for models like DeepSeek-V3.2 and GLM-5.
- Hardware Optimized: Specifically tuned for NVIDIA B200 clusters using a pre-built binary wheel and Docker image for strict ABI compatibility.