vLLM Day-0 Support for Recursive's Flint

vLLM has integrated day-0 support for Recursive's Flint, a 40B parameter Mixture-of-Experts (MoE) model, allowing users to serve the model efficiently on a single GB200 GPU with high throughput.

Flint Model Overview and Capabilities

Flint is a 40B-parameter MoE model that activates approximately 2.5B parameters per token. It is designed with a large context window of 262K tokens and is available in two checkpoints: NVFP4 (quantization-aware trained) and BF16.

The NVFP4 checkpoint is provided as a GGUF file, which vLLM repacks losslessly at load time. This process allows the model to run weight-only (W4A16) while maintaining BF16 activations and the KV cache, ensuring accuracy matches Recursive's reference implementation across benchmarks like GSM8K, GPQA-Diamond, long-context MRCR, and a tool-calling suite.

Flint Model Architecture

Flint utilizes a 49-layer pre-norm MoE architecture with a hidden size of 2048. Its unique design emphasizes local attention to manage the large context window efficiently.

  • Attention Mechanism: Most layers (35 layers) use a 513-token sliding window with RoPE for attention. Only 14 layers attend to the full context without positional encoding. This design limits the KV cache cost to 28 KiB per token, as only the 14 full-context layers retain the entire sequence's KV state.
  • MoE Structure: The model features 256 experts, employing a sigmoid top-6 selection mechanism plus a shared expert. Layers 0 and 1 utilize a dense MLP instead of the MoE structure.
  • Attention Sinks: The architecture incorporates learned attention sinks, uses Grouped Query Attention (GQA) with 32 query and 4 KV heads, and employs per-head Q/K RMSNorm.

Performance and Serving Implementation

Serving Flint on a single GB200 GPU achieves a decode rate of about 500 tokens/s, even when processing a 100K token context. This performance is the result of extensive kernel-level optimizations implemented within vLLM.

Flint support is available via the day-0 Docker image vllm/vllm-openai:flint (supporting x86_64 and arm64 with CUDA 13.0).

Usage Examples:

For the NVFP4 checkpoint (GGUF), the Docker command is:

docker run --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:flint \
  RecursiveSI/flint-nvfp4-gguf \
  --hf-config-path RecursiveSI/flint --tokenizer RecursiveSI/flint \
  --enable-auto-tool-choice --tool-call-parser glm47 \
  --reasoning-parser glm47 --exclude-tools-when-tool-choice-none

For the BF16 checkpoint, the Docker command is:

docker run --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:flint \
  RecursiveSI/flint \
  --enable-auto-tool-choice --tool-call-parser glm47 \
  --reasoning-parser glm47 --exclude-tools-when-tool-choice-none

Optimization Strategies

The performance gains were achieved by replacing stock vLLM layers with specialized Flint kernels, which were rigorously checked against the reference implementation.

Key optimizations include:

  • Small-batch MoE Kernels: These kernels allow the router and expert GEMVs to read only the selected experts' weights, resulting in a 27% reduction in decode step latency.
  • Fused Kernels: The fusion of Q/K norm, RoPE, and KV-cache write into a single kernel reduced latency by 8%.
  • Low-Latency Decode Path: For up to 8 requests, the low-latency path fuses MoE layer operations (Q/K norm, RoPE, routing, and expert combine) into neighboring GEMV kernels. This path reduces NVFP4 decode time by 2.7× (from 4.81 ms to 1.78 ms) and BF16 decode time by 1.5× (from 3.53 ms to 2.34 ms).

Serving end-to-end, batch-1 decode at 100K context takes 2.06 ms per token, which includes the forward pass, LM head, and sampling.

Sources