tile-ai/TileRT

Tile-Based Runtime for Ultra-Low-Latency LLM Inference

What it solves

TileRT is designed for ultra-low-latency LLM inference, specifically targeting scenarios where the time per output token (TPOT) must be minimized for responsiveness. It addresses the limitations of traditional inference systems that prioritize high-throughput batch processing over the latency of individual requests, making it suitable for real-time applications like high-frequency trading, interactive AI, and AI-assisted coding.

How it works

TileRT uses a tile-level runtime engine and a compiler-driven approach to decompose LLM operators into fine-grained tile-level tasks. The runtime then dynamically reschedules computation, I/O, and communication across multiple devices in a highly overlapped manner to minimize idle time and maximize hardware utilization. It also supports Multi-Token Prediction (MTP) to generate multiple tokens per forward pass, reducing sequential decoding depth.

Who it’s for

It is intended for developers and organizations deploying massive models (hundreds of billions of parameters) who require millisecond-level responsiveness on high-end hardware, specifically NVIDIA B200 GPUs.

Highlights

  • Extreme Performance: Capable of breaking 1000 tokens/s on a 1-trillion-parameter model on a single 8-GPU node.
  • PD Disaggregation: Supports prefill-decode disaggregation, allowing vLLM to handle the prefill phase while TileRT manages the decode phase via an OpenAI-compatible endpoint.
  • MTP Support: Integrates Multi-Token Prediction to increase decoding rates.
  • Model Support: Specifically optimized for DeepSeek-V3.2 and GLM-5/5.1 models.

Related

  • Project
    NVIDIA/TensorRT-LLM

    An open-source optimization library from NVIDIA that maximizes inference performance, throughput, and efficiency for LLMs and visual generation models on NVIDIA GPUs.

  • Dispatch
    vLLM and TileRT Integration for Latency-Critical Serving

    vLLM has integrated TileRT 0.1.5 as a pluggable decode engine to provide native per-user decode speed for latency-critical workloads while maintaining vLLM's standard prefill and serving infrastructure.

  • Project
    alibaba/rtp-llm

    A high-performance LLM inference acceleration engine developed by Alibaba that optimizes throughput and latency for production-scale deployments.

  • Project
    triton-inference-server/tensorrtllm_backend

    A Triton Inference Server backend that enables high-performance serving of TensorRT-LLM optimized models with support for in-flight batching and multi-GPU scaling.

  • Project
    google-ai-edge/LiteRT

    Google's high-performance on-device runtime for deploying ML and Generative AI models across mobile, web, and IoT platforms with advanced hardware acceleration.