Speculative decoding is the cheapest throughput lever in production inference: a small draft model proposes tokens, the big model verifies them in one forward pass. The operational catch has always been that the draft model is fixed at engine startup. Tuning it — swapping drafts, updating weights, testing a new checkpoint — meant a full restart. On a busy inference box that means draining the KV cache, dropping the prefix cache, and eating CUDA-graph re-capture. vLLM v0.26.0, released 27 July, removes that cost with runtime draft weight updates (#46725).

The Restart Was the Real Cost of Tuning Drafts

Before this release, changing the draft model meant SIGTERM, re-spawn the server, wait through model loading and warmup, then slowly repopulate the prefix cache as clients retry. For Qwen-class deployments where a qwen-eagle or MTP drafter is tuned per workload, that friction alone discouraged experimentation. Runtime draft weight updates let you swap the drafter while the engine keeps serving. That changes the tuning loop from "restart and observe" to "hot-swap and A/B."

The release notes don't spell out an endpoint or flag in the snippet, but the PR lands the capability in the engine core's speculative-decoding path. Combined with the rest of the speculative batch in v0.26.0, this is the most interesting section of the release notes for anyone serving with drafts:

  • Hybrid (SWA + full attention) DFlash drafters (#47914) — sliding-window drafters can now cover full-attention targets without a separate architecture.
  • SWA support for qwen-eagle3 (#47568) — the default drafter for Qwen3.5-class models gets sliding-window attention support.
  • Gemma4-12B DSpark draft model (#47216) — a 12B drafter for the DSpark path.
  • A separate kv_cache_dtype for speculative_config (#48787) — you no longer need the draft model's KV cache at the same precision as the target, which matters when the main cache is fp8 or MXFP4.
  • TPOT optimization for thinking budget when used with speculative decoding (#46662) — directly relevant if you've seen reasoning models waste tokens on thinking when MTP is on.

The Rest of the Release Worth Knowing

The release is big: 411 commits from 212 contributors, 61 of them new. Beyond speculative decoding, three things are worth reading twice.

First, the DeepSeek-V4 performance push across vendors: a specialized routing kernel claims 2.94% end-to-end TPOT improvement (#48660), fused_topk_bias gives a 1.5–2x kernel speedup (#47463), and redundant repeat/copy removal adds 1.8% TPOT (#48137). If you serve DeepSeek-V4, these are free tokens per watt.

Second, fp32 lm_head for generation models via head_dtype (#48390), extended to the LoRA path (#48525) with a ROCm torch.mm fast path (#48688). Accuracy for generation heads without forcing the whole model to fp32.

Third, KV offloading matured past the experimental stage: an object-store secondary tier with workload identity (#47063), DP-replica-aware tiering (#47987), offloading metrics (#45958), and encoder-cache connectors with CPU offloading (#42433, #47423). If you're pushing context length past what fits in VRAM, this is the release where tiered KV storage stops being a gamble. The attention backend can also now be selected per KV-cache group (#48012), which matters for hybrid SWA models.

What I'd Do

Upgrade and make the runtime draft update the reason. If you're already running qwen-eagle or DFlash drafters, retuning them per workload was previously a scheduled-maintenance event. Now it's an operational tweak. The catch: hot-swapping weights still requires knowing your acceptance rate. A drafter that drifts from the target model silently burns compute on rejected tokens — the tooling just makes the iteration loop fast enough that you can actually measure it.

And check kv_cache_dtype inside speculative_config. Keeping the draft's KV cache at lower precision than the main cache is a clean memory win, and v0.26.0 finally decouples the two. That alone can buy back the VRAM a second drafter costs.

Sources: