Speculative Decoding
Each decode step of a large model reads every weight from HBM to produce one token, and most tokens ("the", "of", closing brackets, boilerplate) don't need 70B parameters to predict. Speculative decoding lets a small draft model propose several tokens cheaply, then has the large target model verify all of them in a single forward pass. Accepted tokens cost one target step for many tokens; rejected ones fall back to normal decoding. The remarkable part: with the right acceptance rule, the output distribution is mathematically identical to sampling from the target model alone.
The algorithm
One cycle of speculative decoding with draft model , target model , and lookahead :
- Draft: run the small model autoregressively for steps, sampling proposals with probabilities .
- Verify: run the target model once on the prefix plus all proposals. Because a transformer scores every position in parallel, this single pass yields for all — plus the distribution for position .
- Accept/reject left to right. For each proposal :
- accept with probability ;
- on the first rejection, discard and everything after it, and sample the replacement token from the residual distribution .
- Bonus token: if all proposals are accepted, sample token directly from the target's already-computed distribution — free.
Each cycle therefore emits between (immediate rejection, resample) and (all accepted + bonus) tokens for one target forward pass.
Why the output distribution is exactly
The accept/resample rule is a form of rejection sampling. For any token , the probability it is emitted at a given position is
The first term equals . The rejection mass sums to (both equal half the total variation distance), which cancels the residual's normalizer, so the second term equals . Their sum is exactly .
No approximation, no distribution shift — a worse draft model only makes speculative decoding slower, never wrong. This is why it's classed as a quality-neutral lever in the inference-optimization decision tree.
Worked trace
Vocabulary , lookahead . The draft proposes the → cat → sat, and the target's verification pass returns:
| Position | Proposal | Draw | Outcome | |||
|---|---|---|---|---|---|---|
| 1 | the | 0.80 | 0.72 | 0.90 | 0.41 | ✅ accept |
| 2 | cat | 0.60 | 0.66 | 1.00 | — | ✅ accept |
| 3 | sat | 0.70 | 0.14 | 0.20 | 0.55 | ❌ reject |
Position 3 rejects, so we resample from the residual there. Suppose at position 3: and over (the, cat, sat, mat). Then , which normalizes to — the resample almost surely emits mat, the token the draft badly under-proposed. The cycle emits the, cat, mat: 3 tokens for one target pass instead of three.
Watch it run
The cycle above, decision by decision, and then the check that matters:
Speculative decoding — draft, verify, and the rule that keeps it exact
A 4-token vocabulary and lookahead k = 3. The small draft model proposes "the → cat → sat" for a few percent of the cost of one target step, and the target then scores all three positions in a **single** forward pass — a transformer computes every position in parallel, so verifying 3 tokens costs the same as generating 1.
The bottleneck in decoding is memory bandwidth, not arithmetic: one target step reads every weight from HBM to produce one token. Producing several tokens per read is the entire idea.
One draft-and-verify cycle on the worked example above: two proposals accepted, one rejected, and a resample from the residual max(0, p − q) that lands on the token the draft under-proposed. The final step is the important one — 200 000 emissions under a well-matched draft and a deliberately terrible one, both reproducing the target distribution to four decimal places. A bad draft costs speed and nothing else.
What to notice as you step:
- Under-proposing is free. At position 2 the target likes "cat" more than the draft did, so and no random number is even drawn. Only over-confidence by the draft can cause a rejection.
- A rejection discards everything after it, because those proposals were conditioned on a token that will not be emitted. This is why acceptance rate compounds, and why lookahead has diminishing returns.
- The residual is a correction, not a fallback. puts its mass exactly where the draft was under-confident, which is what makes the emitted distribution come out equal to rather than merely close to it.
- The last step verifies that claim empirically: 200 000 emissions under a well-matched draft (94% acceptance) and a deliberately terrible one (26%), both reproducing to four decimal places. A worse draft costs throughput and nothing else.
Expected speedup
Let be the average per-token acceptance probability (empirically 0.7–0.9 for a well-matched draft). The expected number of tokens emitted per cycle is the expectation of a truncated geometric plus the bonus token:
If the draft costs a fraction of a target step, the wall-clock speedup per emitted token is
At , , : per cycle and speedup . Diminishing returns in are steep — decays fast, so beyond ~4–8 mostly burns draft compute. Note the acceptance rate also depends on sampling temperature: at low temperature concentrates, agreement rises, and speculation wins more.
Variants
| Variant | Draft source | Trade-off |
|---|---|---|
| Draft–target (classic) | Separate small model (same tokenizer) | Best acceptance; must host a second model |
| Medusa | Extra decoding heads on the target itself | No second model; heads need finetuning; tree-verifies multiple branches |
| EAGLE | Lightweight head predicting from the target's hidden state | Higher acceptance than Medusa at similar cost; current default in vLLM/TRT-LLM |
| Prompt lookup / n-gram | Copy matching spans from the prompt | Free; only wins on extractive/repetitive tasks (RAG quotes, code edits) |
| Self-speculative | Target with layers skipped | No extra weights; modest acceptance |
When it wins and loses
- ✅ Templated, predictable text (support answers, code boilerplate, JSON) acceptance rates soar.
- ✅ Low-batch, latency-sensitive serving — spare compute exists between bandwidth-bound decode steps; verification rides it free.
- ❌ Large batches — the GPU is already compute-saturated, and verifying positions per sequence competes with other requests; the speedup shrinks toward zero.
- ❌ High-entropy generation (creative writing at high temperature) — acceptance drops and cycles waste draft work.
Related concepts
- Optimizing LLM Inference: where speculative decoding sits among the model-level levers
- Inference Optimization & Serving: why decode is bandwidth-bound, which is what speculation exploits
- Decoding & Sampling Strategies: the sampling procedures the acceptance rule must preserve