Speculative Decoding

Speculative Decoding

Speculative decoding is one of the cleverest optimization techniques in modern AI. It uses a small, fast model to help a large model generate text more quickly. The basic idea is simple: let the small model guess what the large model would say, then have the large model verify the guesses all at once.

Normally, a large model generates one token at a time, and each token requires loading all the model weights from memory. This is slow because of the memory bandwidth bottleneck we discussed earlier. But if the large model could generate multiple tokens at once, it would be much faster. The problem is that the large model cannot predict multiple tokens ahead without seeing the intermediate tokens.

Speculative decoding solves this by using a small draft model, maybe 10x smaller than the main model, to generate several candidate tokens. The draft model is fast because it is small. Then the large model processes all the candidate tokens at once, in parallel, and checks which ones it agrees with. Since the draft model is reasonably good, most of its guesses are correct, and the large model accepts them.

The key insight is that verifying a batch of tokens is almost as fast as generating one token, because the weights only need to be loaded once for the batch. So if the draft model generates 5 tokens and the large model accepts all of them, you have effectively generated 5 tokens in the time it would normally take to generate 1 token. This gives a 5x speedup when the draft model is perfectly accurate.

In practice, speculative decoding achieves 2x to 3x speedups on real workloads. The speedup depends on how well the draft model matches the large model. If the draft model is too different, most of its guesses will be rejected, and the speedup is minimal. But with a well matched draft model, speculative decoding is a free lunch: faster generation without any loss of quality.