vLLM TT Plugin brings Tenstorrent accelerators to LLM serving

TL;DR

The vLLM TT Plugin adds Tenstorrent accelerator support to the vLLM serving stack, preserving the same OpenAI‑compatible API while introducing a mesh‑native scheduler, on‑device sampling, and a single‑process lane data‑parallel design.

Overview of the TT Plugin

The plugin is distributed as an out‑of‑tree platform module that activates automatically when the ttnn package from TT‑Metal is importable. No changes to client code or request format are required; the OpenAI‑compatible API works unchanged.

Supported Model Families

The plugin registers Tenstorrent‑backed architectures with a TT prefix. Model selection is based on architecture, not model name, allowing a single registration to cover multiple releases. The current list includes:

Model family TT architecture class
Llama 3.1 / 3.2 / 3.3 TTLlamaForCausalLM
Llama 3.2 Vision TTMllamaForConditionalGeneration
Qwen 2.5 / Qwen 3 TTQwen2ForCausalLM, TTQwen3ForCausalLM
Qwen 3.5 / 3.6 TTQwen3_5ForConditionalGeneration
Qwen 2.5‑VL / 3‑VL TTQwen2_5_VLForConditionalGeneration, TTQwen3VLForConditionalGeneration
Mistral / Mistral 3 TTMistralForCausalLM, TTMistral3ForConditionalGeneration
Gemma 3 TTGemma3ForConditionalGeneration
Gemma 4 TTGemma4ForCausalLM, TTGemma4ForConditionalGeneration, TTGemma4UnifiedForConditionalGeneration
DeepSeek V3 TTDeepseekV3ForCausalLM
GPT‑OSS 20B / 120B TTGptOssForCausalLM

Multimodal models such as Llama 3.2 Vision, Qwen‑VL, Qwen 3.6, Mistral 3, and Gemma 3 are already served via the plugin.

Architecture‑Centric Registration

The plugin does not contain model code; it merely registers architecture names. The actual implementations live in TT‑Metal, where each class wraps a hand‑written TTNN model. Because registration matches on architecture, a single class can serve multiple model releases (e.g., TTQwen3_5ForConditionalGeneration serves Qwen/Qwen3.6-27B).

Custom models can be added without modifying the plugin source by pointing EXTRA_MODELS_DIR at a directory containing a vllm_metadata.json and an adapter class. Setting TT_VLLM_BUILTIN_MODELS=0 limits the registry to user‑supplied models only.

Tenstorrent Mesh vs. GPU‑Shaped Inference Stacks

Tenstorrent hardware is a mesh of cores and chips connected by an on‑fabric network (e.g., n150, n300, QuietBox, Galaxy). Programs are compiled against a fixed mesh shape, and data movement between chips is baked into the compiled trace rather than issued as host‑side collectives.

Key consequences of this compilation model:

  • No tensor‑parallel or pipeline‑parallel ranks – the mesh program encodes parallelism directly. The MESH_DEVICE=TG flag replaces the usual --tensor‑parallel-size argument, and the plugin rejects -tp/-pp.
  • Step granularity is a whole traced program – each step replays a captured trace for a fixed batch shape, making homogeneous batches much cheaper than heterogeneous ones.
  • Sampling can be performed on device – the mesh program can return the selected token directly, eliminating host‑side logits transfer.

These differences required substantial adaptations in vLLM’s plugin interfaces.

Plugin Integration Points

The vLLM hardware‑plugin mechanism (introduced May 2025) provides two entry points:

Entry point group Name Target
vllm.platform_plugins tt vllm_tt_plugin.entrypoints:platform_plugin
vllm.general_plugins tt_model_registry vllm_tt_plugin.entrypoints:register

platform_plugin() returns a TTPlatform instance only when ttnn is importable, preventing accidental activation in pure CUDA environments.

The plugin swaps in Tenstorrent‑specific runtime classes via vLLM’s extension points:

vLLM config field TT implementation
parallel_config.worker_cls vllm_tt_plugin.worker.TTWorker
scheduler_config.scheduler_cls vllm_tt_plugin.scheduler.TTScheduler or vllm_tt_plugin.lane_scheduler.TTLaneCoordinator

Device‑specific options are passed through vLLM’s generic --additional-config namespace, e.g.:

--additional-config.tt.sample_on_device_mode all
--additional-config.tt.fabric_config FABRIC_1D_RING

No Tenstorrent‑specific code resides in vLLM core, ensuring forward compatibility with upstream releases.

Phase‑Based Scheduling

Unlike vLLM’s token‑budget scheduler, the Tenstorrent path restricts each scheduling step to one of three homogeneous outcomes:

  1. prefill‑only
  2. decode‑only
  3. empty

Mixed prefill‑and‑decode batches are not allowed. Long prompts are split into multiple prefill‑only steps, with decode‑only steps interleaved to keep other requests progressing. This design preserves trace stability, which is essential for compiled mesh programs.

What the Phase Split Gains

  • Enables reuse of a single compiled trace per step shape.
  • Mirrors the “disaggregated serving” pattern used in large GPU fleets, but applied within a single engine.

What It Costs

  • Decode requests wait for each prefill chunk, incurring a step‑level latency penalty.
  • The scheduler must switch modes between steps, which adds a small policy overhead.

The design remains extensible; future versions could capture mixed‑shape traces if needed.

Single‑Process Lane Data Parallelism on Galaxy

Galaxy (a 32‑chip mesh) runs some models as a single‑execute program spanning the entire mesh. Traditional multi‑process data parallelism cannot be applied because there is only one mesh submission.

The solution is in‑process lane DP:

  • TTLaneCoordinator creates one TTScheduler per lane (four lanes by default).
  • Each lane maintains its own waiting/running queues, KV cache, and block‑ID space.
  • Requests are assigned to the least‑loaded lane and stay bound to it.
  • At each step the coordinator selects a shared mode (prefill or decode) for all lanes. Lanes without work contribute empty slices.
  • The merged batch is sent to the device once; results are split back to lanes internally.

This eliminates the costly inter‑process scatter/gather that plagued an earlier multi‑process attempt.

Edge Cases

If a prefill step admits zero tokens due to KV pressure while another lane has decode work, the step is retried in decode mode to avoid a deadlock.

User‑Facing Flags

The usual vLLM flags are reused:

MESH_DEVICE=TG \
TT_LLAMA_TEXT_VER=llama3_70b_galaxy \
VLLM_RPC_TIMEOUT=900000 \
python examples/server_example_tt.py \
  --model "meta-llama/Llama-3.3-70B-Instruct" \
  --data_parallel_size 4 \
  --max_num_seqs 8 \
  --async-scheduling \
  --additional-config.tt.dispatch_core_axis col \
  --additional-config.tt.sample_on_device_mode all \
  --additional-config.tt.fabric_config FABRIC_1D_RING \
  --additional-config.tt.worker_l1_size 1344544 \
  --additional-config.tt.trace_region_size 220000000

--data_parallel_size 4 now creates four in‑process lanes, each capable of handling --max_num_seqs requests.

On‑Device Sampling with Automatic Fallback

When sample_on_device_mode is set, the mesh program performs token selection and returns tokens directly. If a batch requires features the device cannot express (logprobs, penalties, custom logits processors, etc.), the plugin falls back to vLLM’s host‑side sampler for that batch only. The always_compat_sampling flag forces host‑side sampling for debugging.

Asynchronous Decode Overlap

The plugin offers decode/host overlap, but only as asynchronous host readback, not as a separate device execution thread:

  1. Submit decode work without blocking (read_from_device=False).
  2. Initiate non‑blocking host readback (async_read=True).
  3. Store the resulting events.
  4. At finalization, synchronize with ttnn.event_synchronize() before converting to host tensors.

An in‑flight queue of depth 2 allows the host to schedule step N+1 while step N’s readback is still in progress. Overlap is maintained only for steady‑state generation (stable shape, on‑device sampling, no structured‑output bookkeeping). Prefill remains synchronous.

Current Limitations

The plugin validates configurations early and rejects unsupported combinations:

  • Tensor‑parallel and pipeline‑parallel are expressed via mesh shape, not vLLM ranks.
  • Speculative decoding, LoRA, and prompt logprobs are not yet supported.
  • Prefix caching is available only for models that declare it.
  • Async decode overlap requires a model‑declared capability.
  • Standard multi‑process DP does not support MoE models; lane‑DP is used instead.
  • Multi‑host serving is not implemented.

These are limitations of the current TT‑Metal runtime and model implementations, not hard hardware constraints.

Getting Started

  1. Install TT‑Metal following its official guide.
  2. Clone and install the plugin:
    git clone https://github.com/tenstorrent/vllm-tt-plugin.git
cd vllm-tt-plugin
source docs/install-vllm-tt.sh
    The script builds vLLM against version 0.26.0 inside the TT‑Metal environment.
  3. Serve a model:
    MESH_DEVICE=T3K VLLM_RPC_TIMEOUT=100000 python examples/server_example_tt.py
  4. Query via any OpenAI‑compatible client, e.g.:
    curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-70B-Instruct", "prompt": "San Francisco is a", "max_tokens": 32}'

Roadmap

  • Expand async‑decode support to more model families.
  • Enable prefix caching for additional models and lane‑DP RoPE handling.
  • Implement speculative decoding once mesh‑side draft/verify pipelines are stable.
  • Add multi‑host serving to scale beyond a single machine.

Acknowledgements

The plugin builds on the vLLM platform‑plugin mechanism contributed by the Ascend team and the pluggable‑scheduler design from the Spyre team. Thanks to the vLLM maintainers for keeping the extension points generic enough for a mesh architecture.

Contributors include Viktor Puš, Tomasz Cheda, Sanjar Adylov, and Salar Hosseini. Feedback on the lane‑DP user surface and priority model families is welcomed via GitHub issues or the vLLM Slack.

Sources