vLLM Constrained Decoding Fixes Tool-Call Accuracy to 100% for Kimi and DeepSeek

Takeaway

Enabling the constrained decoding flag introduced in vLLM 0.31 guarantees 100 % tool‑call schema adherence for Kimi K2.6 and DeepSeek V4 Flash, eliminating the malformed calls that previously reduced accuracy to 92 %.

Why Tool‑Call Schema Accuracy Matters

  • A tool call must follow the JSON schema advertised to the model; any deviation causes the call to fail and pollutes the context window.
  • In multi‑turn agentic workloads, a single malformed call forces retries, raises latency, and degrades overall model quality.
  • Moonshot AI’s K2VV ToolCall benchmark measures two key metrics:
    • Trigger similarity – how often the service decides to invoke a tool compared to the reference implementation.
    • Schema accuracy – the fraction of generated calls that conform to the declared schema. A 50 % schema accuracy means half of all calls are unusable.
  • When Ubicloud served Kimi K2.6 with vLLM 0.28, schema accuracy was only 92 % (2 failures per 25 calls), which is unacceptable for production agents.

Constrained Decoding: How It Works

  • Standard decoding lets the model sample any token based on probability, which can produce syntactically invalid JSON.
  • Constrained decoding restricts the token set at each generation step to only those that keep the output valid according to the tool’s JSON schema.
  • vLLM implements this using XGrammar structural tags. The grammar defines a deterministic sequence:
    1. <|tool_call_begin|>{function.name}:{index}
    2. <|tool_call_argument_begin|>
    3. JSONSchemaFormat – generates JSON that conforms to the supplied schema.
    4. <|tool_call_end|>
  • Invalid tokens are masked on‑the‑fly, and the mask is pre‑computed for each phase, so the GPU can apply it with negligible overhead.

"This changes schema accuracy from a best‑effort approach to a runtime guarantee. The model still decides which token to generate, but now the token is selected from a restricted and syntactically‑correct set." – Ubicloud blog

What Changed in vLLM 0.31

  • Constrained decoding was added as a server‑side flag (--tool-strict-level) in vLLM 0.31.
  • Prior to 0.31, the feature existed in earlier releases (e.g., 0.25) but was disabled by default for most models to avoid regressions on models that do not follow the expected tool‑calling format.
  • Enabling the flag for Kimi and DeepSeek forced the engine to apply the XGrammar masks, raising schema accuracy to 100 % in the K2VV benchmark.

Why the Default Was Disabled

  • vLLM serves a wide variety of models; some models (e.g., gpt‑oss‑20b, Qwen 3.6, GLM 5.1) generate tool calls that diverge from the expected format, causing constrained decoding to reject too many tokens and degrade performance.
  • Three categories of issues were identified:
    1. General model issues – models stray from the expected schema when unconstrained.
    2. Grammar translation bugs – errors in converting a model’s tool schema into XGrammar tags.
    3. Feature interaction bugs – e.g., conflicts between constrained decoding and speculative decoding that caused valid tokens to be masked incorrectly.
  • Because of these regressions, vLLM’s maintainers chose a model‑agnostic default of disabled constrained decoding and introduced the explicit --tool-strict-level flag for users who need strict enforcement.

Community Insight

"Constrained decoding has been in vLLM for a long time, and constrained tool‑call decoding has been in vLLM for a while, too. The blog post is rather confusing because until you understand the entire situation it appears to imply that they fixed vLLM (what the title implies) but really they just needed to configure it correctly." – @Aurornis (HN comment)

  • The discussion highlights that the real fix was configuration, not a code change, and credits Ubicloud for maintaining a benchmark that caught the regression.
  • Other commenters note that constrained decoding is now common in other runtimes (e.g., llama.cpp, HuggingFace transformers.js) and that 100 % tool‑call accuracy is a baseline expectation for production agents.

Practical Guidance for Deployments

  1. Enable the flag when serving models that have a stable tool‑calling format (e.g., Kimi K2.6, DeepSeek V4 Flash):
    vllm serve model_path --tool-strict-level=2   # or the appropriate level per vLLM docs
  2. Validate with a benchmark such as Moonshot AI’s K2VV ToolCall to ensure schema accuracy remains at 100 % after any engine upgrade.
  3. Monitor for model‑specific regressions; if a model shows degraded performance with constrained decoding, consider disabling the flag for that model only.
  4. Keep XGrammar definitions up‑to‑date when adding or renaming tools; the grammar must reflect the exact JSON schema used by the model.

Conclusion

Tool‑call schema accuracy is a critical determinant of agentic LLM quality. By turning on vLLM 0.31’s constrained decoding, Ubicloud eliminated malformed tool calls, raising benchmark accuracy from 92 % to 100 % for Kimi and DeepSeek models. This demonstrates that inference‑engine configuration can be as important as model selection for delivering reliable, high‑quality AI services.

Sources