One llama.cpp flag made Qwen 3.8 3x faster

MTP kept showing up in my comments. Multi-token prediction. It had been on in every benchmark I published, at depth 2, because Unsloth's launcher turns it on without saying so, and if you had asked me what it did I would have had no real idea. That is not a comfortable place to be for someone who posts benchmark numbers, so this video is me going down the rabbit hole and then measuring the thing on my own desk.

The setup: Qwen 3.8-27B at full BF16 (a 55 GB file) on my 128 GB mini PC, MTP off versus draft depths 2, 3 and 4, three deliberately different prompts, and the whole sweep run twice under two protocols. Experiment 1 uses the model card's recommended sampling with thinking on. Experiment 2 is the clean room: thinking off, greedy decoding, fixed seed, so if the output changes, only MTP could have changed it.

What MTP actually does

A language model writes one token at a time, and every single token means reading the whole model out of memory once. On a 55 GB model that memory trip is the speed limit; the arithmetic finishes long before the read does, so the compute mostly waits. MTP gives the model a scribbling assistant: Qwen 3.8 ships with a tiny extra head trained to guess the next few tokens cheaply, and the full model then checks all of those guesses in a single one of those expensive memory trips. Guesses it agrees with are kept, so one trip can pay out two, three or four tokens. The first wrong guess and everything after it is thrown away, and the model carries on from there exactly as it would have anyway.

The lineage is short: speculative decoding with a separate draft model (Google, 2023), then Meta training the draft heads into the model itself (2024), then DeepSeek V3 shipping it as standard. llama.cpp got support in July 2026, one flag and the head is already inside the GGUF you downloaded.

The numbers

The headline is 3.09x, Experiment 2 Boilerplate at depth 4, 4.18 to 12.91 tokens per second on the same machine. The shape is what the theory predicts: the more predictable the next few tokens, the more guesses survive, the bigger the win. Code and formula-driven tables sit near 95% acceptance and triple. Free-form prose falls to 33 to 53% acceptance, still faster than off, but deeper drafts start costing time there, and the Essay at depth 4 in Experiment 1 took twelve minutes against nine at depth 2.

Two bonus measurements. The marathon: the Table prompt at Qwen's factory-default xhigh reasoning effort, a 12,000-token thinking workload, went from 47.7 minutes with MTP off to 20 minutes at depth 2. And memory: the draft head costs about a gigabyte of peak system memory, which on a 128 GB unified-memory box is noise.

Does it change the output?

This is the part I actually wanted to see. The promise of speculative decoding is that the output is the same as the base model, only faster. In Experiment 2, with the dice removed, that held: the Table and the Boilerplate came back byte-identical at every depth, verified by hash, and the Boilerplate compiles. The one exception is the Essay at depths 3 and 4, which flipped even at temperature 0, because prose is full of near-ties so close that the numeric difference between the two decoding paths alone can swap the top word. The llama.cpp maintainers describe exactly this: different kernels for different batch sizes, expected, not a bug.

Experiment 1, with thinking and sampling on, forked. The three MTP depths agreed with each other on the Boilerplate and the odd one out was MTP off, the plain model. I could not tell from one run whether that meant anything, so I reran the MTP-off leg on a different day with the identical configuration. It reproduced itself byte for byte on all three prompts, thinking transcript included. So the fork is not run-to-run noise. Same seed, same model, one variable changed, and that variable is the decoding path. Neither leg is "right"; both are valid samples of the same model, and the clean room shows they agree exactly once sampling is off.

Every raw output, thinking transcript, hash, timestamp and side-by-side split is on the full results page, so you can check my work instead of trusting it.

What I would do with it

This applies to my machine, my prompts and my two protocols, and I am sharing it for people curious what I ran into. For a model like Qwen 3.8: check whether the flag is on in your llama.cpp or launcher, because the default may not be what you think (Unsloth turns it on at depth 2 and does not print it; older llama.cpp builds may not have it at all), and have it on at depth 2 for every kind of task. Depths 3 and 4 are something to test for your own use case: on this build they added more on coding and structured output and less on creative writing, and the trade-off depends on your hardware and memory.

Honest limit: my acceptance rates beat the published Strix Halo numbers, and the main reason is that my prompts were written to be predictable. That is prompt design, not a hardware result, and it is a skill I am still building. The better move is to run the established benchmark suites and contribute to them, so the numbers are comparable instead of merely mine.

Sources

• Donato Capitella, MTP on Strix Halo, the explainer this builds on, measured on my hardware family
• Devsplainers, Multi-Token Prediction explained, the memory-bandwidth framing and the MoE, concurrency and small-model caveats
• kyuz0, Strix Halo MTP benchmarks, same platform, Qwen 3.6
• llama.cpp PR #22673, MTP support, and the output-difference threads #23302 and #23335
• Leviathan et al. 2023, speculative decoding and Gloeckle et al. 2024, multi-token prediction
• DeepSeek-V3 technical report
• Qwen 3.8-27B model card and the Unsloth GGUF files
• My earlier runs where MTP was on without my knowing: the quant showdown and the thinking-levels test

Creating in public, figuring out tech together. This one took a few days of overnight runs and a rerun I almost did not do; the rerun turned out to be the best part.