Abstract

Speculative decoding accelerates autoregressive language model inference by having a small draft model propose several tokens at once, which the target model then verifies in parallel. The gain is real: on commodity hardware, a well-tuned draft-target pair can cut wall-clock latency by 2-3x with no change in output distribution. The field has moved fast since the original Google and DeepMind papers in 2023, and current work explores replacing the single draft model with ensembles, learned drafters, and self-speculative approaches where the target model drafts from its own early layers.

The Core Mechanism and Its Bottleneck

Standard speculative decoding runs the draft model autoregressively for k steps, producing a candidate token sequence, then passes all k positions to the target model in one forward pass. The target’s parallel logits either accept or reject each draft token via a modified rejection sampling scheme that preserves the exact target distribution. The acceptance rate, the fraction of draft tokens the target keeps, determines the speedup. Low acceptance rates collapse the gain; a draft model that produces plausible but subtly off-distribution tokens relative to the target can perform worse than no speculation at all. Acceptance rate is the core engineering metric.

Ensemble Drafting

One direction for improving acceptance rate is running multiple small draft models and selecting the candidate that maximizes agreement with a proxy of the target distribution before sending anything for verification. The overhead of running N drafters in parallel is small if they are tiny (sub-1B parameter) models and the target is large (70B+), because the verification step dominates cost. Research from MIT and independent teams has shown that even simple ensemble strategies, such as taking the token on which the highest fraction of drafters agree, improve acceptance rates by 5-15 percentage points on code and math domains where the target model is highly peaked. The tradeoff is memory bandwidth, since each drafter occupies KV cache.

Self-Speculative and Medusa Variants

A different approach avoids a separate draft model entirely. Medusa, from together.ai, attaches additional decoding heads to the target model that predict tokens at future positions given the current hidden state. These heads are fine-tuned separately and run cheaply at inference time. The EAGLE approach refines this by conditioning the speculative heads on the draft token embeddings as well, substantially improving acceptance rates on instruction-following tasks. Self-speculation from early exit layers, explored in work from Meta and Tsinghua, uses the target model’s own intermediate representations to draft, trading model memory for a cleaner distribution match. As of early 2026, Medusa and EAGLE implementations are available in llm.cpp and vLLM, making them accessible without custom CUDA kernels.