vLLM Adaptive Verification Without a Confidence Head: Online-Fitted Acceptance Estimator (PR #52228)
TL;DR
vLLM now supports confidence scoring for all draft-model-based speculator types, not just DSpark drafters that ship with a trained confidence head. When no confidence head exists, an online-fitted estimator derives a feature from the drafter's logits — the log-odds of the drafter's most likely token — and uses it to predict the target model's probability of accepting each draft token. That prediction is used directly as the confidence score for Adaptive Verification (AV). The estimator's coefficients are continuously refitted during live serving, so it adapts to the incoming data traffic. The feature landed in PR #52228 and is enabled with enable_adaptive_verification for draft models without a confidence head.
What Adaptive Verification Is and Why It Needed a Confidence Score
Adaptive Verification verifies only a subset of drafted tokens during speculative decoding, trimming draft tokens with a low probability of acceptance from the batch before the target model's forward pass runs. This saves compute, and it is particularly beneficial while serving large batch sizes, because in that regime the GPU becomes compute-bound.
AV requires a per-token estimate of acceptance probability. Until this change, that requirement restricted AV to DSpark drafters, which often ship with a confidence head: a small linear layer over the drafter's hidden state for each position, trained as a binary classifier for "will the target accept this token?"
Draft models that lack a confidence head — EAGLE, MTP, DFlash, and some DSpark checkpoints — could not use AV at all. Adding a confidence head would mean retraining the drafter, which is not an option for the many existing checkpoints already being served.
The Solution: A Logistic Regression Fitted During Serving
vLLM added a lightweight logistic regression that estimates each draft token's acceptance probability from a single scalar derived from the draft distribution: the log-odds of the drafter's most likely token. The model has one shared slope and one intercept per draft position, so a K-token drafter needs a total of K + 1 parameters. Those parameters are fitted online from the accept/reject decisions the target model already produces during normal verification. Any existing draft checkpoint gets AV for free.
The confidence score for a given draft token is estimated with a per-position logistic regression, where ω is the fitted slope shared across all draft positions, βi is the fitted per-position intercept, and q is the draft distribution. In log-odds form, logit(qmax) is the estimator's feature — the log-odds of the draft's most likely token, representing how confident the draft model is in its prediction. That feature is transformed linearly by ω and βi to predict the log-odds of the draft's acceptance by the target model. The sigmoid of that value is p, the estimated acceptance rate, which is used as the confidence score for AV.
The Four Criteria Used to Choose a Feature
Four axes mattered most when evaluating candidate features:
- Losslessness — whether the feature biases the output distribution during speculative decoding.
- Area Under the Curve (AUC) — the probability that a randomly chosen accepted draft scores higher than a randomly selected rejected one. This measures discrimination between drafts that should be accepted versus rejected; 0.5 is random chance.
- Log-loss — the mean negative log-likelihood of the predicted acceptance probability, which penalizes confident wrong answers and rewards calibration.
- Expected Calibration Error (ECE) — bin each draft by predicted acceptance probability, then measure the error between the mean prediction and the mean observed frequency within each bin, weighted by bin occupancy. This measures calibration.
Losslessness is non-negotiable: without it, speculative decoding loses its guarantee of outputting the target model's exact distribution. AUC and ECE both matter for AV's trimming decision but in different ways. AUC governs the local ordering of survival probabilities, ensuring a more likely-to-be-accepted draft token is admitted over a less likely one. ECE governs the global budget decision, where predicted probabilities are summed and divided by a measured cost table, so ECE ensures the estimated number of accepted tokens matches reality.
Feature Comparison Results
The candidate features were measured over roughly 4.5M labeled drafts collected while serving DeepSeek-V4-Flash + DSpark (temp=1, top-p=0.95, conc=64), without AV, across five datasets: SPEED-Bench 2k low/high-entropy, GSM8K, HumanEval, and MT-Bench. The last two rows are references rather than fitted features.
| feature | lossless | AUC | log-loss | ECE |
|---|---|---|---|---|
| log(qmax) | yes | 0.8069 | 0.4813 | 0.0592 |
| log(qmax) − log(q2nd) (top-2 margin) | yes | 0.8156 | 0.4437 | 0.0105 |
| H(q) (entropy) | yes | 0.8321 | 0.4577 | 0.0638 |
| −log(−log(qmax)) | yes | 0.8336 | 0.4289 | 0.0103 |
| *logit(qmax) (shipped)* | yes | 0.8336 | 0.4284 | 0.0100 |
| logit(qsampled) | no | 0.8553 | 0.4040 | 0.0152 |
| −log(−log(qsampled)) | no | 0.8554 | 0.4075 | 0.0193 |
| qmax (no fit, used directly as the probability) | yes | 0.8259 | 0.4880 | 0.0736 |
| trained confidence head | yes | 0.8191 | 0.4449 | 0.0052 |
logit(q_sampled) and −log(−log(q_sampled)) clearly outperform the chosen feature on AUC, but were rejected because trimming based on them produces a biased output distribution. Both read qsampled, the probability of the token the drafter actually drew. Trimming a draft on a function of that value makes the decision to verify at position i dependent on xi itself, so surviving drafts are no longer distributed as q. Standard rejection sampling accepts a token with probability min(1, p/q), expecting q to be the distribution the draft token was sampled from; trimming on qsampled biases the emitted distribution toward the drafter's higher-confidence tokens.
The qmax "no fit" row explains why fitting is needed at all, instead of using the drafter's own probability directly as EAGLE-2 does when building its draft tree. Used directly, the drafter's top-token probability ranks drafts better than the trained confidence head (0.8259 vs 0.8191 AUC), but it is not well calibrated: its mean prediction over the labeled drafts is 0.784 against an observed acceptance rate of 0.723, and its ECE is 7x worse than the shipped logit(qmax) feature (0.0736 vs 0.0100). The drafter's top-token probability reflects only how confident the drafter is in its own prediction, not whether the target model will agree — and that agreement is further influenced by the sampling setup and the workload. An online fit learns this mapping directly from the target's accept/reject decisions, yielding better calibration, which is what the budget decision depends on.
Estimator Versus a Trained Confidence Head
On DeepSeek-V4-Flash + DSpark, the estimator outperforms the confidence head in both AUC and log-loss, but underperforms in ECE.
| AUC ↑ | log-loss ↓ | ECE ↓ | |
|---|---|---|---|
| estimator logit(qmax) | 0.8336 | 0.4284 | 0.0100 |
| trained confidence head | 0.8191 | 0.4449 | 0.0052 |
| Δ (estimator − head) | +0.0146 | −0.0166 | +0.0048 |
Once the first draft position is excluded from the averages, the picture changes substantially: the estimator's AUC advantage widens to +0.0278 and its log-loss advantage to −0.0323, while the ECE gap narrows to +0.0013, leaving the head and estimator virtually tied on calibration.
| AUC ↑ | log-loss ↓ | ECE ↓ | |
|---|---|---|---|
| estimator logit(qmax) | 0.8352 | 0.4420 | 0.0103 |
| trained confidence head | 0.8075 | 0.4743 | 0.0090 |
| Δ (estimator − head) | +0.0278 | −0.0323 | +0.0013 |
The per-position AUC breakdown explains the shift. Measured on DeepSeek-V4-Flash + DSpark at 7 speculative tokens over 4.47M labeled drafts, the confidence head leads at the first position, but the estimator leads from the second position onwards, and the gap widens with depth — the estimator leads by 0.08 AUC by the last position. Both predictors fall off after the fifth position, likely because DSV4-Flash's DSpark declares a block size of 5 and the drafter was never trained past it, so the drafts themselves get worse there.
Because survival is a running product and the first draft position always carries a request's highest score, it is less often the token the budget cuts. The estimator's advantage sits exactly where trimming happens, which explains why it can match the confidence head in end-to-end performance.
How the Coefficients Are Fitted
The coefficients ω and βi are continuously refitted during live serving. Draft tokens proposed during the previous decode step are verified by the target model in the next decode step, and only draft tokens up until the first rejection (if any) are eligible for training, because the target model has explicitly labeled them accept or reject.
vLLM uses Iteratively Reweighted Least Squares (IRLS) to minimize the log-likelihood loss of the predictions against the labels, minimizing over a second-order Taylor approximation of the loss. The optimization step for ω and βi is obtained by solving a system of equations in which Δθ is the delta for the shared slope and per-position intercepts, H is the Hessian of the loss function with shape (K + 1) × (K + 1), and g is the gradient of the loss function with size K + 1.
A refit happens after every fixed round of N decode steps. During each decode step in that interval, the Hessian and gradient values are accumulated from the coefficients, features, and labels collected during the round. This upper-bounds memory to O(K²) across a round, in contrast with O(N × B × K) if raw labels were collected instead. On the final step of the round, the system is solved and the coefficients are updated.
Deeper drafts are accepted less often than shallower drafts, so deeper draft positions can be starved of labels during a round. Two design decisions mitigate this:
- Shared slope — the slope is shared across all draft positions to prevent noisy updates caused by data thinning at depth. Predictions are much more sensitive to slope errors than intercept errors, so sharing makes the Newton step well-determined. Empirical measurements show sharing the slope costs virtually nothing in accuracy: only 0.0002 AUC and 0.0003 log-loss.
- Step damping — each coefficient's update is scaled by the amount of supporting evidence using the formula n / (n + constant), where n is the count of graded drafts. Deeper positions with fewer labels therefore move more conservatively, preventing updates based on noise.
Slope and Intercepts Play Different Roles
Holding the model pair fixed and varying the workload: fitting DSV4-Flash + DSpark independently on each of five workloads, the slope barely moves (0.51 to 0.55) and is uncorrelated with how hard the workload is (r = −0.08 against mean draft entropy). The intercepts do the opposite — the first draft position intercept correlates with draft entropy at r = −0.89, running from ~0.5–0.6 on the more predictable workloads down to below 0 on the least.
| Dataset | Acceptance Rate | Mean Draft Entropy | Slope ω | Intercept β0 |
|---|---|---|---|---|
| gsm8k | 0.781 | 0.562 | 0.5365 | +0.478 |
| humaneval | 0.751 | 0.669 | 0.5135 | +0.486 |
| speed-bench 2k low-entropy | 0.755 | 0.703 | 0.5087 | +0.610 |
| mt-bench | 0.714 | 0.800 | 0.5495 | +0.181 |
| speed-bench 2k high-entropy | 0.642 | 1.077 | 0.5197 | −0.089 |
Varying the model pair instead: the slope ranges from 0.45 to 0.59 across pairs while staying tight within each one. The intercepts also differ between pairs, so unlike the slope they are not a property of one axis alone.
| Model | nspec | Slope ω | ω sd | Intercept β0 |
|---|---|---|---|---|
| DeepSeek-V4-Flash + DSpark | 5 | 0.5013 | 0.0250 | +0.279 |
| MiMo-V2.5-Pro-FP4 + DFlash | 8 | 0.5197 | 0.0415 | −1.000 |
| Inkling-NVFP4 + MTP | 8 | 0.5859 | 0.0349 | −0.572 |
| Muse-Glimmer-30B + DFlash2 | 7 | 0.4450 | 0.0323 | +0.167 |
Read together, the two tables suggest a division of labor. The slope measures how well the draft model knows when it is right — a property of the target/draft pair and the draft sampling method, which is why it barely moves when only the workload changes. The intercepts absorb the rest: the base acceptance rate at each draft position, which falls with depth and drops further on less predictable content. That base acceptance rate depends on the model pair as well as the workload, which is why the intercepts move along both axes while the slope moves along just one.
Benchmarks
Three checkpoints that previously could not use AV at all were benchmarked — MiMo-V2.5-Pro-FP4 + DFlash, Inkling-NVFP4 + MTP, and Muse-Glimmer-30B + DFlash2 — along with DSV4-Flash + DSpark + the estimator and DSV4-Flash + DSpark + the confidence head as a reference. Gains were captured with AV off versus on, temperature 1.0, top-p 0.95, and concurrency 64, over four datasets.
Compared to fixed verification, the online estimator at the same speculative length is a win across all benchmarked datasets. That gain alone does not tell the whole story, however: a smaller fixed length can often be found that outperforms AV at the max speculative length at specific concurrencies. A Pareto curve captured with DSV4 + DSpark across a sweep of fixed speculative lengths and AV at the max speculative length shows the online estimator and the confidence head roughly on par in performance across all measured concurrencies.
The online estimator is not meant to be a replacement for the confidence head, but rather a drop-in alternative in the absence of one. On MiMo-V2.5-Pro-FP4 + DFlash, a model pair that currently lacks a confidence head, AV sits on the throughput-interactivity frontier at 5 of 7 concurrencies.
Limitations
- The target model's attention backend must support variable-length verification. This is the same prerequisite that DSpark AV already has.
- MTP and EAGLE benefit less from AV than DFlash and DSpark do, because those auto-regressive speculators pay a proportionally larger cost for drafting tokens that may get trimmed before verification.
How to Enable It in vLLM
AV is enabled through the speculative config. On a draft model that has no confidence head and supports variable length attention for all of its backends, the online estimator is selected automatically:
vllm serve <target_model> \
...
--speculative-config '{
"model": "<draft_model>",
"method": "<method>",
"num_speculative_tokens": <num_speculative_tokens>,
"draft_sample_method": "probabilistic",
"enable_adaptive_verification": true
}'
Prior Work and Acknowledgements
Using the drafter's own confidence as a proxy for acceptance is well established. EAGLE-2 showed that EAGLE's draft probabilities closely track acceptance rates and used them to grow and rerank a dynamic draft tree. Draft & Verify and several follow-ups stop drafting once the drafter's probability falls below a fixed threshold. SpecDec++ instead trains a dedicated acceptance-prediction head, similar in spirit to DSpark's confidence head. In each case, the mapping from draft confidence to acceptance is either used as-is or learned offline. AV needs more than a good ranking: its budget decision sums predicted acceptance probabilities and weighs them against measured costs, so predictions must be calibrated for the specific model pair and workload being served.
This work was done by Giancarlo Delfin (Inferact), and builds directly on Adaptive Verification by Lucas Wilkinson (Red Hat) and Benjamin Chislett (NVIDIA).
Why Losslessness Rules Out the Highest-AUC Features
An example with a simple two-token vocabulary of [a, b] illustrates the bias introduced by trimming on qsampled. Assume p = [0.5, 0.5], q = [0.2, 0.8], and that a draft token is trimmed from verification if q(x) < 0.5. Drawing a results in trimming, and the target samples a token from its own distribution, p(x). Drawing b results in verification: the token is accepted with probability min(1, p(b)/q(b)) = 0.625, and otherwise rejected and resampled from the residual norm(max(0, p − q)) = [1, 0], yielding a. During rejection sampling this gives pout(a) = q(a)·p(a) + q(b)·(1 − 0.625) = 0.4 and pout(b) = q(a)·p(b) + q(b)·0.625 = 0.6. That differs from the target's distribution of [0.5, 0.5] and violates speculative decoding's lossless guarantee.