A new way to speed up token prediction
Adaptive Multi-Token Prediction can make decoding up to 4.1x faster.
Researchers created Adaptive Multi-Token Prediction, or AMTP. This method can make decoding up to 4.1x faster than standard decoding.
Multi-token prediction, or MTP, drafts several tokens and verifies them in one pass. This can accelerate decoding.
The best speculation horizon changes with context. A longer draft offers greater amortization when reliable, but can waste compute if suffixes are discarded after a rejection.
AMTP uses a lightweight controller to pick a horizon. It uses verification feedback to learn how far a draft can be trusted in each context.
The controller predicts how far a draft is likely to survive. It picks the horizon with the highest expected tokens per second.
In tests using Llama-3.1-8B-Instruct with Medusa heads, Meta's MTP models, and Qwen3-4B-Inst-2507-MTP, AMTP matched or outperformed fixed windows in 8 of 10 settings. It showed throughput gains of up to 10%.
Key facts
- AMTP decodes up to 4.1x faster than standard decoding.
- AMTP provides throughput gains of up to 10%.
- The controller requires no retraining and leaves the exact verifier unchanged.