§24.1 and §24.2 covered post-training quantization — train in fp16, then quantize for deployment. That gets you to ~4 bits with modest perplexity drop. Past 4 bits, PTQ degrades sharply and you need a different approach: quantization-aware training (QAT) puts the quantizer inside the training loop. The forward pass uses quantized weights; gradients flow through as if the quantizer didn’t exist (the straight-through estimator). The network learns weights that are quantization-friendly. This unlocks 2-bit and even 1-bit weights with much smaller quality loss. The same trick — STE through a non-differentiable operation — also powers BitNet (1.58-bit models trained from scratch) and QLoRA (the technique that lets you fine-tune Llama 2 70B on a 24GB consumer GPU). This section closes the chapter by walking the math, the kernel, and the production techniques.

The straight-through estimator (STE)

The quantization operation Q(W) = scale · round(W / scale) has zero derivative almost everywhere (it’s a step function) and undefined derivative at the step boundaries. Plain backprop through it would produce no gradient. The straight-through estimator QAT mechanism A technique for backpropagating through a non-differentiable operation (typically quantization or the sign function) by pretending in the backward pass that the operation is the identity. Forward: Y = Q(W). Backward: ∂L/∂W = ∂L/∂Y (just pass the gradient through unchanged). Introduced by Bengio 2013 for binary networks; the central tool of every modern QAT method. Surprisingly effective despite being a 'wrong' gradient — empirically the loss landscape with STE behaves well enough to train. sidesteps this by pretending in the backward pass that Q is the identity:

The STE is “wrong” — Q’s true derivative isn’t 1. But it’s wrong in a way that works empirically: the gradient still points in roughly the right direction (small perturbations of W produce small perturbations of Q(W) on average, even though pointwise it jumps), and the network learns weights that, when quantized, give good outputs.

The fake-quantize-in-forward + STE-in-backward pattern is the entire idea behind QAT training method A training method that puts the quantizer inside the training loop. Each forward pass uses fake-quantized weights (W_q = scale · round(W / scale)); the backward pass uses the straight-through estimator to flow gradients through as if Q were the identity. The fp32 'shadow' weights W continue to be updated normally; their quantized form W_q is what the network actually uses for computation. Result: weights settle into configurations that work well after quantization. Used to recover quality at 4-bit and below. . The kernel below trains a 2-layer MLP two ways and measures the deployment cost:

/* Symmetric int4 fake-quantize (per-block of QK) */
static void fake_quantize_int4(const float* W, float* W_fq, int n) {
    for (int b = 0; b < n; b += QK) {
        int end = b + QK; if (end > n) end = n;
        float amax = 0;
        for (int i = b; i < end; i++) {
            float a = fabsf(W[i]); if (a > amax) amax = a;
        }
        float scale = amax / 7.0f;
        if (scale == 0) scale = 1.0f;
        for (int i = b; i < end; i++) {
            int q = (int)roundf(W[i] / scale);
            if (q >  7) q =  7;
            if (q < -8) q = -8;
            W_fq[i] = q * scale;
        }
    }
}

/* Generate a synthetic regression dataset: y = sin(sum(x)) + noise. */

Output:

Mode 1: train in fp32 (no QAT)
  fp32  ep   0  loss = 0.79713
  fp32  ep 100  loss = 0.37293
  fp32  ep 199  loss = 0.34471

Mode 2: train with QAT (forward pass uses Q(W) via STE)
   qat  ep   0  loss = 0.80807
   qat  ep 100  loss = 0.37556
   qat  ep 199  loss = 0.34672

--- Final eval ---
Model              fp32 deploy   int4 deploy   gap
trained fp32       0.34449       0.35018       +1.65%
trained QAT (STE)  0.34799       0.34424       -1.08%

Read carefully: the fp32-trained model has loss 0.344 in fp32 deployment, 0.350 in int4 — a 1.65% degradation. The QAT-trained model has slightly worse fp32 loss (0.348) but essentially zero degradation when deployed in int4 (0.344). On the deployment metric (int4), QAT wins. It traded a tiny bit of fp32 quality for a model whose weights are pre-adapted to be quantized.

Learned Step Size Quantization (LSQ)

A refinement: instead of using a fixed scale per block, make the scale a learnable parameter. The optimizer trains both the weights and the per-block scale. Esser 2019 “Learned Step Size Quantization”:

LSQ lets the model discover the optimal scale per block during training, instead of using the absmax heuristic. Typical gains: 0.2-0.5 perplexity over absmax-based QAT at the same bpw. Standard in modern QAT pipelines for sub-4-bit quantization.

BitNet — 1-bit (and 1.58-bit) from scratch

Wang 2023 “BitNet” took QAT to its extreme: 1-bit weights, trained from scratch.

A 1-bit weight matrix has no per-element scale and only one sign bit per weight. The matmul Y = X · W becomes a sequence of additions and subtractions — no multiplications. On custom hardware, this is dramatically cheaper than even int8 matmul.

Ma 2024 “BitNet b1.58” refined this to three states 1 — 1.58 bits per weight (log₂ 3 ≈ 1.585). The third “0” state lets the network gate connections, which empirically recovers most of the quality gap to fp16. BitNet b1.58 reportedly matches fp16 Llama at sizes ≥ 3B params on standard benchmarks, with 8× memory reduction and similar compute reduction on hardware that exploits the ternary structure.

BitNet b1.58 ternary network A neural network with ternary weights ∈ {-1, 0, +1}, requiring ~1.58 bits per weight (log₂ 3 = 1.585). Trained from scratch with STE through the ternary quantizer. Matches fp16 Llama performance at parameter counts ≥ 3B on standard benchmarks (Ma 2024). The matmul becomes a sequence of additions and subtractions (no multiplies), which on custom hardware translates to dramatic compute and energy savings. The most ambitious 'quantize from scratch' result to date. is the strongest evidence that quantization-aware training can fundamentally change the cost / quality frontier — not just compress an already-trained model, but produce a fundamentally cheaper model that’s as good.

QLoRA — the production trick

The most impactful application of QAT-adjacent techniques is QLoRA fine-tuning method A fine-tuning method (Dettmers 2023) that combines 4-bit quantized base weights with fp16 LoRA adapters. The base model is loaded in 4-bit NF4 (a special quantization data type derived from the normal distribution); during fine-tuning, the base weights are frozen and dequantized on-the-fly to fp16 only for the matmul; small LoRA adapter matrices A, B ∈ fp16 are added to the matmul output. Only A and B are trained. Memory: a 70B base model in 4-bit fits in ~40 GB; LoRA adapters add ~0.5 GB. Allows fine-tuning 70B-parameter models on a single 24 GB consumer GPU. — Dettmers 2023 — the technique that put fine-tuning of 70B models within reach of solo researchers.

The data type used for the base is NF4 QLoRA data type A 4-bit floating-point-like data type optimised for normally-distributed weights. Instead of evenly-spaced quantization levels (q4_0's int4), NF4 uses 16 levels positioned at the quantiles of a standard normal distribution. For zero-mean Gaussian-distributed weights (which LLM weights approximately are), NF4 achieves lower quantization error than int4 at the same 4-bit cost. The default base-weight data type in QLoRA. (NormalFloat 4-bit), which uses 16 quantization levels positioned at the quantiles of a normal distribution. Since LLM weights ARE approximately normally distributed, NF4 represents them more accurately than evenly-spaced int4. The choice is matched to the empirical weight distribution.

Why this works as well as fine-tuning the full fp16 model:

1. The LoRA adapters B · A have full fp16 precision, so they can represent any necessary correction to W_base.
2. The frozen 4-bit base is “good enough” — its quantization error is in directions the LoRA adapter can correct.
3. The base + adapter combination at inference is effectively a higher-precision matrix than 4-bit alone.

The memory math is brutal in QLoRA’s favor. A 70B model:

QLoRA is the workhorse of single-GPU LLM customization. Every “fine-tune your own Llama” tutorial on Hugging Face goes through QLoRA.

The picture across the chapter

Three layers of quantization, three different scopes:

Where each lives in production:

• PTQ (q4_K_M, AWQ, GPTQ): every llama.cpp deployment, every Hugging Face quantized model. Default for inference.
• QAT: research and bespoke deployments, increasingly mainstream for 2-3 bit production targets. Apple’s foundation models reportedly use it.
• BitNet b1.58: still mostly research as of 2026, but Microsoft’s 100B-class BitNet 4.5 model is in active development and may ship.
• QLoRA: every fine-tune-your-own tutorial; every research project that customises a 70B model; the entire “fine-tune Llama” ecosystem on Hugging Face.

END OF CH.24 — Quantization.
§1 (PTQ basics, blockwise vs per-tensor, outliers, LLM.int8 → GPTQ → AWQ) · §2 (the GGML family: q4_0 to q6_K with exact byte layouts, q4_K_M model naming, IQ-quants + imatrix) · §3 (QAT + STE, BitNet b1.58, QLoRA — the workhorses of low-bit training and single-GPU fine-tuning).

Coming next: back to the normal chapter order. Ch.16 — Pretraining, Chinchilla scaling laws, the token budget.