The Quant Ladder

A model is just a giant list of numbers, ~27 billion of them here. Quantization only changes how carefully each number is written down. BF16 stores every number with 16 bits, like writing pi as 3.14159265: precise, and fat (55 GB). Q8 uses 8 bits per number (3.1416, you rounded, but barely). Q4 uses 4 bits (3.14, still basically pi). Q2 uses 2 bits ("3-ish", now you are gambling).

Best analogy: the same photo saved at different JPEG quality. RAW (BF16), high-quality JPEG (Q8, nobody can tell), normal JPEG (Q4, fine unless you zoom), and a meme reposted forty times (Q2, blocky, but you still know it is a cat). Rounding billions of numbers means billions of tiny errors; the question this page answers is whether those errors add up to anything you can actually see. Bonus: the "UD" in Unsloth's file names means dynamic, the important numbers keep more bits and the boring ones get squeezed hardest, which is why a good 4-bit beats a naive 4-bit.

All five ladder runs, instruct mode, thinking OFF: temperature 0.7 · top_p 0.8 · top_k 20 · presence_penalty 1.5 (exactly the official card's recommended non-thinking settings) · seed 42 · max output 20,000 tokens · one prompt, one shot, no retries · 60-min limit per test · tools disabled everywhere (Unsloth: --disable-tools; Ollama: no tools sent, Ollama executes nothing server-side). Unsloth legs disable thinking via chat_template_kwargs: {"enable_thinking": false} with a 65,536-token context; the Ollama leg via the native API's think: false with min_p 0, repeat_penalty 1 and a 32,768-token context. Note: Ollama ships different sampler defaults (temperature 1.0, top_p 0.95); those were overridden to the settings above so the only variable is the build and quant, not sampler luck. "Thinking found in output: none" means verified in the outputs, not just configured. Thinking mode is not on this page at all; it gets its own video with the card's official thinking settings.

Every token requires reading all the model's weights from memory. At full precision that is ~55 GB per token; this chip's memory bus peaks at 256 GB/s on paper (256-bit LPDDR5X-8000, ~215 GB/s measured in practice), which caps you in single-digit tok/s. Quantization shrinks the gigabytes each token has to read: half the bytes, roughly double the speed. That is the whole ladder: 55 GB → 9 tok/s, 29 GB → 17, 18 GB → 20, 11 GB → 28. The question this video answers is what quality each step down costs.

The file only holds the model's weights. At runtime you also need: the KV cache, the model's working memory of everything in the current conversation, it grows with context length (a couple of GB at a moderate context, potentially much more at 32k-64k); compute buffers, scratch space for the math of each token; and the runtime's own overhead (a few hundred MB). That is why "fits" depends on how big a context you run, and why a model that barely fits will suddenly spill the day you paste a long document into it.