vLLM and TileRT Integration for Latency-Critical Serving

vLLM and TileRT Integration for Latency-Critical Serving

vLLM has integrated TileRT 0.1.5 as a pluggable decode engine, allowing users to combine vLLM's prefill capabilities with TileRT's specialized decode speed. This integration enables latency-critical workloads—such as real-time voice, interactive coding assistants, and agentic loops—to achieve maximum per-user token generation speed without sacrificing the operational maturity, APIs, and scheduling of the vLLM ecosystem.

Pluggable Decode Architecture

By leveraging disaggregated serving—the separation of the compute-bound prefill phase from the memory-bandwidth-bound decode phase—vLLM allows the decode side to become pluggable. The integration of TileRT is achieved through vLLM V1's public connector interface (KVConnectorBase_V1), meaning the TileRT decode pool can be added to a deployment without modifying the vLLM source code or creating a fork.

Key Components of the Integration

  • vLLM Prefill: Handles scheduling, chunked prefill, and prefix caching.
  • vLLM Serving Surface: Maintains OpenAI-compatible APIs, request formats, and existing tooling.
  • TileRT Decode: A specialized inference runtime designed to push per-user decode speed to hardware limits.
  • MultiConnector: Allows a single stock vLLM prefill pool to serve both native vLLM decode pools (for high-throughput) and TileRT decode pools (for low-latency).

Request Routing and Handoff

To maintain coexistence between native and specialized decode pools, the system uses a lightweight router and a specific claim filtering mechanism.

Routing Mechanism

For latency-critical traffic, the router sets max_tokens=1 so that vLLM performs the prefill and emits the first token. It then attaches target decode node information via kv_transfer_params (specifically tilert_host and tilert_ctrl_port). General traffic continues to flow through the standard disaggregation proxy unmodified.

Data Plane and State Transfer

The handoff from vLLM prefill to TileRT decode is designed to be fast and non-blocking:

  • RDMA Transfer: Attention state (compressed KV, sparse-attention index caches, and metadata) is moved via RDMA one-sided writes into pre-registered GPU buffers using either Mooncake or NIXL.
  • Prefill Overlap: State extraction occurs within the forward window; state is copied to a staging buffer before cache blocks are recycled, and a background sender handles the network transfer. This ensures that TileRT-bound requests do not block subsequent prefill iterations.
  • Immediate Execution: Upon arrival, state is converted to TileRT's native layout and injected into the running engine, where multi-token speculative decoding begins immediately.

Performance and Use Case Selection

Choosing between decode pools depends on the specific requirements of the workload:

  • Use TileRT Decode: When per-user token speed is the primary constraint (e.g., interactive agents, latency-SLO inference) and the model is supported.
  • Use Native vLLM Decode: When maximum aggregate throughput, high-concurrency batching, and broad model/feature support are required.

Current Limitations and Support

As of TileRT 0.1.5, a TileRT decode node serves one in-flight request at a time, with the router managing gated dispatch and back-pressure. Supported models currently include GLM-5/5.1 and DeepSeek-V3.2.

Implementation and Setup

Deploying the TileRT pool requires installing TileRT 0.1.5 via PyPI or GitHub. The process involves three primary steps:

  1. Weight Conversion: Converting Hugging Face checkpoints to TileRT's weight format using the weight_converter module.
  2. Decode Server Launch: Starting the decode_server with specific configurations for MTP (Multi-Token Prediction) and KV cache dtype.
  3. vLLM Prefill Configuration: Running vllm serve with the TileRTConnector plugin specified in the --kv-transfer-config, ensuring the speculative-config is set to mtp so the prefill node populates the draft-layer KV for the decode-side speculation.

Sources