tile-ai/TileRT

Tile-Based Runtime for Ultra-Low-Latency LLM Inference

What it solves

TileRT is designed for ultra-low-latency LLM inference, specifically targeting scenarios where the time per output token (TPOT) must be minimized. While traditional systems focus on high-throughput batch processing, TileRT prioritizes responsiveness for real-time applications like high-frequency trading, interactive AI, and long-running agents.

How it works

TileRT uses a tile-level runtime engine and a compiler-driven approach. It decomposes LLM operators into fine-grained "tile-level" tasks and dynamically reschedules computation, I/O, and communication across multiple devices to overlap these operations. This minimizes hardware idle time and improves utilization. It also supports Multi-Token Prediction (MTP), allowing the model to generate multiple tokens in a single forward pass to reduce sequential decoding depth.

Who it’s for

It is intended for developers and organizations deploying massive models (hundreds of billions of parameters) on high-end hardware (such as NVIDIA B200 or AMD Instinct MI350X/MI355X) who require millisecond-level responsiveness for single-user generation.

Highlights

  • Extreme Performance: Capable of breaking 1000 tokens/s on a 1-trillion-parameter model on a single 8-GPU node.
  • Hardware Support: Optimized for NVIDIA B200 and AMD Instinct MI350X/MI355X GPUs.
  • MTP Integration: Built-in support for Multi-Token Prediction to accelerate decoding rates.
  • PD Disaggregation: Supports prefill-decode disaggregation, allowing vLLM to handle the prefill phase while TileRT manages the decode phase via an OpenAI-compatible endpoint.

Related

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    flashrt-project/FlashRT

    A high-performance real-time inference engine optimized for small-batch, latency-sensitive AI workloads across NVIDIA, AMD, and Ascend hardware.

  • Dispatch
    vLLM and TileRT Integration for Latency-Critical Serving

    vLLM has integrated TileRT 0.1.5 as a pluggable decode engine to provide native per-user decode speed for latency-critical workloads while maintaining vLLM's standard prefill and serving infrastructure.

  • Project
    alibaba/rtp-llm

    A high-performance LLM inference acceleration engine developed by Alibaba that optimizes throughput and latency for production-scale deployments.

  • Project
    triton-inference-server/tensorrtllm_backend

    A Triton Inference Server backend that enables high-performance serving of TensorRT-LLM optimized models with support for in-flight batching and multi-GPU scaling.