vLLM Day-0 Support for Inkling 2 Small: 594B MoE, 1M Context, and 630 TPS/User

vLLM provides day-0 support for Thinking Machines' Inkling 2 Small, a multimodal text-and-vision MoE model with 594B total parameters, 17B active parameters, and up to 1M tokens of context. The integration matters because vLLM ships optimized serving for the model at launch, including mixed-precision MoE execution, a native DSpark speculator, LoRA support, and kernels tuned for long-context agentic workloads.

TL;DR

  • 594B total, 17B active, 1M context. Inkling 2 Small is a 48-layer MoE with 384 experts (top-8) that takes images and text and outputs text.
  • Architecture upgrade from Inkling. It adds block attention residuals, attention sink, gated attention, and a ViT image encoder.
  • Mixed-precision MoE. The MoE uses NVFP4 in layers 0–39 and MXFP8 in layers 40–47, so the model can be served with a single node.
  • Native DSpark speculator. DSpark with an RNN head drafts 7 tokens per step.
  • Extensive optimizations. Kernel fusion, PDL, a tuned Gluon attention kernel, and sequence parallelism achieve up to 630 TPS/user on GB200 at TP4 with acceptance length 3.5.
  • RL and LoRA support. vLLM supports RL via SkyRL and LoRA for the model.

Serving command

The model can be served with vLLM using tensor parallelism, the Inkling reasoning and tool-call parsers, automatic tool choice, and DSpark speculative decoding:

vllm serve thinkingmachines/Inkling-2-Small \
  --tensor-parallel-size 8 \
  --reasoning-parser inkling --tool-call-parser inkling --enable-auto-tool-choice \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'

Model architecture

Inkling 2 Small keeps Inkling's backbone and adds newer architectural components. The backbone includes a 5:1 mix of sliding-window and full attention, GQA with a learned relative bias instead of RoPE, four short convolutions per layer with kernel size 4, and a sigmoid-routed MoE.

The model has 48 layers: 40 sliding-window layers and 8 full-attention layers. It uses 48 query heads and 8 KV heads in all layers, with a sliding window of 1,024 tokens. The MoE has 384 routed experts with top-8 routing, no shared experts, and MoE in all layers.

Compared with Inkling, the architecture changes are substantial:

Component Inkling Inkling 2 Small
Layers 66 (55 sliding-window + 11 full) 48 (40 sliding-window + 8 full)
Residual path residual stream block attention residuals (8 blocks of 6 layers)
Attention GQA with relative bias, no RoPE same
Heads (query / KV) 64 / 8 full, 64 / 16 sliding 48 / 8 in all layers
Sliding window 512 1,024
Attention output plain sigmoid gate and per-token sink
Short convolutions 4 per layer (K, V, attention out, MLP out), kernel size 4 same
MoE 256 routed (top-6) + 2 shared expert sinks, first 2 layers dense 384 routed (top-8), no shared experts, all layers MoE
Expert precision NVFP4 NVFP4 (layers 0–39), MXFP8 (layers 40–47)
Modality text, image, audio in; text out text, image in; text out
Image encoder 4-layer hMLP on 40×40 patches 23-layer ViT (1,024 wide) on 20×20 patches
Speculative decoding 8 chained MTP heads DSpark draft, 7 tokens per step

Block attention residuals replace the residual stream. Before each sublayer, a learned query mixes the outputs of earlier 6-layer blocks. Gated attention puts a sigmoid gate on each head's output, together with a per-token attention sink. The image encoder is a 23-layer ViT, 1,024 wide, operating on 20×20 patches, replacing Inkling's 4-layer hMLP encoder on 40×40 patches.

Mixed-precision MoE with Four-Over-Six NVFP4

Inkling 2 Small uses mixed expert precision across its 48 layers. The experts in the first 40 layers use NVFP4, and the experts in the last 8 layers use MXFP8. This mixed-precision design allows the model to be served with a single node.

The NVFP4 activations use a per-token global scale plus dual-candidate block scales, a scheme called Four Over Six. For each 16-element block, the quantizer tries scaling the block's maximum to 6 and to 4, then keeps whichever candidate gives the lower reconstruction error measured by MAE.

DSpark speculator with an RNN head

The checkpoint ships with a DSpark draft in place of Inkling's 8 chained MTP heads. The speculator has 5 layers and drafts 7 tokens per step. Its head is the RNN variant described in DeepSeek's DSpark paper.

vLLM integration and optimization

vLLM's integration targets the model's specific bottlenecks: memory-bound sublayer boundaries, MoE activation quantization, redundant prefill work under tensor parallelism, and slow decode attention caused by the model's GQA ratio.

Kernel fusion and PDL

vLLM fuses collective communication and memory-bound operations between sublayers into a single kernel. With block attention residuals, each boundary needs an all-reduce, a short convolution, a residual update, the attention-residual mix, and an RMSNorm. These are five memory-bound steps on the same hidden states.

The fused kernel, allreduce_conv_mix, runs the all-reduce over NVLink multicast and keeps everything after it in registers. The GEMM, attention, routing, and activation kernels launch with programmatic dependent launch (PDL), so each kernel's prologue overlaps the tail of the previous kernel. This matters at small batch sizes, where a decode step is a long chain of short kernels.

MoE with fused activation quantization

vLLM quantizes expert activations inside the kernels that produce them, matching each layer's NVFP4 or MXFP8 precision. No separate quantization pass touches the hidden states. The routing kernel quantizes tokens before the gate/up projection, and the softcapped SwiGLU quantizes its output before the down projection. The expert GEMMs run on TensorRT-LLM's batched-GEMM kernels, shipped with FlashInfer.

Sequence parallelism for prefill

Under plain tensor parallelism, every GPU runs the short convolution, attention-residual mix, and RMSNorm on all tokens after each all-reduce. The same work is repeated on every rank. This is cheap in decode but costly in prefill.

vLLM removes the redundancy with sequence parallelism. Each GPU reduce-scatters, runs the short convolution, residual update, attention-residual mix, and RMSNorm in one fused conv_mix kernel on only its slice of tokens, then all-gathers the result for the next GEMM. This cuts an 8K-token prefill step by 22% at TP4 and 26% at TP8, and frees 1.3–1.9 GiB per GPU for KV cache.

Optimized Gluon relative attention kernel for decode

Inkling's FlashAttention-4 decode path is slow for Inkling 2 Small because the 6:1 GQA ratio silently disables GQA packing. vLLM wrote a warp-specialized Gluon kernel for decode and verify that packs the 6 heads into one 64-row tile and reads KV once. A TMA warp streams KV, a tcgen05 warp runs the MMAs, and four softmax warps apply the relative bias.

A batch-1 decode step at 100K context drops from 15.1 ms to 4.8 ms. Prefill keeps Inkling's sheared-bias FlashAttention-4 path.

Performance

vLLM benchmarked Inkling 2 Small on an agentic workload at TP8 using two 4-GPU nodes, with DSpark probabilistic drafting and prefix caching. Each session keeps a long context and, on every turn, appends the previous answer and a ~2K-token tool output, so only about 3K tokens per turn are new. Each turn generates 1,024 tokens with reasoning on at temperature 1.0.

The table gives output throughput per GPU at the concurrency where each user gets 100 tokens/s:

Session context Output tok/s per GPU at 100 tok/s/user Concurrent sessions Prefix-cache hit rate Acceptance length
32K ~1,070 ~84 91% 3.3
64K ~915 ~72 95% 3.4
128K ~700 ~55 98% 3.5
256K ~450 ~37 99% 3.5

For a single user at TP4, the model decodes at about 630 tok/s with an acceptance length of 3.5.

Accuracy evaluations

vLLM keeps the reference implementation's numerics: the same data types at every step, FP32 attention output and router, and the same NVFP4/MXFP8 activation quantization. Only the reduction order and the all-reduce's BF16 partials differ.

End to end, vLLM verified accuracy on NIAH up to 1M tokens, MMMU-Pro, and HLE with inkling-verify, with DSpark enabled. The scores match vLLM's runs without speculative decoding.

Implications

Inkling 2 Small is a large sparse multimodal model with a 1M-token context window, and vLLM's day-0 support shows that it can be served with mixed NVFP4/MXFP8 expert precision on a single node. The integration combines architecture-aware kernels, fused communication and normalization, sequence parallelism for prefill, and a custom Gluon decode kernel to make long-context agentic serving practical. vLLM also supports LoRA and RL via SkyRL for the model.

Sources