A Llama 2 70B model in fp16 is 140 GB. At 4-bit weights it’s 35 GB — fits on a single H100. At 2-bit weights it’s 17.5 GB — fits on a consumer 24GB GPU with room to spare. The economics of running an LLM are dominated by this number: bits per weight. The technique that achieves it — replacing high-precision weights with low-bit integers plus a small per-block scale — is called quantization. This section covers the basic linear-map math (scale + optional zero point), the choice of per-tensor vs per-block vs per-channel granularity, the outlier problem that drove LLM.int8 → GPTQ → AWQ, and a kernel that measures the round-trip error of each scheme. The next two sections cover the GGML format family (q4_0 through q6_K, the most-deployed quantization scheme in the world) and quantization-aware training.

The quantization linear map

For a single weight value x ∈ ℝ, the standard quantization map is:

The symmetric form has one parameter (scale) per block; the asymmetric form has two (scale and min). Both are linear maps from a real-valued range onto a discrete integer grid. The cost: b bits per weight for the integer + the per-block fp16 overhead.

Scale quantization parameter A floating-point number (usually fp16) stored once per block of weights that maps the integer representation back to a real value: x ≈ scale · x_q. Chosen so the integer range exactly covers the weight block's range — e.g., scale = max(|x|) / 127 for symmetric int8. The single most-important number in a quantization scheme; getting the scale wrong dominates the quantization error. is the load-bearing parameter. For symmetric int8 with weights in x ∈ [-W_max, W_max], the scale is typically scale = W_max / 127 so the largest weight maps to ±127 (using the full int8 range).

Zero point quantization parameter The integer offset b such that x ≈ scale · (x_q - b). Allows asymmetric quantization where the weight range isn't symmetric around zero. Adds one fp16 per block of overhead but can capture activations with non-zero mean (e.g., ReLU outputs) with one fewer effective bit of error. q4_1 vs q4_0 in GGML differ exactly in whether they store a zero point. is the offset that allows the integer grid to be shifted: x ≈ scale · x_q + min. Asymmetric quantization is more accurate for non-symmetric distributions (e.g., post-ReLU activations always ≥ 0) at the cost of one more fp16 per block.

Per-tensor vs per-block vs per-channel

The granularity vs overhead trade-off is the central one in low-bit quantization. Coarser granularity (per-tensor) saves overhead but suffers from outliers; finer granularity (per-block) eliminates the outlier problem but adds overhead. For int4 (4 bits per weight, the regime most LLM deployments care about), the overhead of per-block scales (0.5 bpw at K=32) is small enough that per-block dominates.

The outlier problem (LLM.int8)

Quantization sounds simple. What broke when people first tried it on real LLMs?

Dettmers 2022 “LLM.int8()” noticed that as transformer models scale past ~6.7B parameters, a small number of “outlier features” emerge in the activations — specific feature dimensions with magnitudes 5-100× larger than the rest. These outliers concentrate in only a few (~6-12) feature dimensions out of d_model = 4096+, but they carry a disproportionate share of the information.

The kernel below demonstrates the outlier problem on synthetic weights. It runs three schemes on the same input (with a small number of injected outliers) and reports the dequantization RMSE and the relative error of the (W · x) dot product:

            if (q >  127) q =  127;
            if (q < -128) q = -128;
            w_dq[i] = q * scale;
        }
    }
}

/* ---- per-block symmetric int4 (q4_0-style symmetric) ------------------- */
static void per_block_int4(const float* w, float* w_dq, int n, int blk) {
    for (int b = 0; b < n; b += blk) {
        int end = b + blk; if (end > n) end = n;
        float absmax = 0;
        for (int i = b; i < end; i++) {
            float a = fabsf(w[i]);
            if (a > absmax) absmax = a;
        }
        /* int4 signed: range [-8, 7]. We use -8 as the negative extreme
           (slightly asymmetric — same as q4_0's choice). */
        float scale = absmax / 8.0f;
        if (scale == 0) scale = 1.0f;
        for (int i = b; i < end; i++) {

Output:

Weight tensor: N = 8192, std = ~0.02, max |w| = 0.335 (after outliers)

scheme                           bits/wt      dequant RMSE  (W·x) rel error
fp32 (baseline)                  32.0         0.000e+00     0.000e+00
per-tensor int8                  8.0          7.533e-04     3.688e-02
per-block int8 (B=32)            8.5          1.282e-04     2.675e-03
per-block int4 (B=32) ≈ q4_0     4.5          2.285e-03     4.768e-02

Three things to read off:

1. Per-tensor int8 has a 3.7% dot-product error. The single outlier set the scale to ~0.335 / 127 ≈ 2.6e-3, but most weights are ~0.02 — so most weights only get ~8 distinct quantization levels. Bad.
2. Per-block int8 is 14× more accurate at almost the same bit count (8.5 vs 8). The outlier’s contamination is confined to one 32-weight block; the other 8160 weights get their own scales.
3. Per-block int4 is comparable to per-tensor int8 (4.8% vs 3.7%) despite using HALF the bits per weight. The per-block scale is doing more work than the extra 4 bits would have.

This is why all production low-bit quantization is per-block. The format that wins isn’t “more bits” — it’s “smaller blocks.”

The PTQ method family

Three landmark post-training quantization methods, in chronological order:

GPTQ PTQ method A post-training quantization method (Frantar 2022) that quantizes one layer at a time using a second-order (Hessian-based) approximation of the loss. For each weight column, GPTQ computes the optimal quantization that minimizes the expected output error given the activation statistics of a small calibration dataset. Achieves 4-bit quantization with ~0.5 perplexity loss on Llama-class models. The standard for 4-bit GPU inference before AWQ. processes weights one column at a time. For each column, it quantizes the column’s entries while simultaneously adjusting later columns to compensate for the quantization error. The adjustment uses the per-layer Hessian (computed from calibration activations) to know which directions are loss-sensitive. Result: 4-bit quantization with only ~0.5 perplexity loss on Llama 7B.

AWQ PTQ method A post-training quantization method (Lin 2023) that uses a small calibration dataset to compute per-channel activation magnitudes, then SCALES weights and inverse-scales activations so that important weight channels (those with large activations) have effectively higher precision. Cheaper than GPTQ (no Hessian, just per-channel statistics) and achieves comparable or better quality. The current default for 4-bit GPU inference. is conceptually simpler: it scans calibration activations and identifies which weight channels are “important” (those that multiply large activations). It then divides those weight channels by a per-channel factor s and multiplies the corresponding activation column by s at inference — the math is unchanged, but the quantization grid now spans the important weights with higher effective resolution. Result: comparable to GPTQ at lower compute cost.

Next: §24.2 — The GGML/llama.cpp quantization family. The exact byte-level layouts of q4_0, q4_1, q5_0, q5_1, q8_0 (legacy formats), q2_K..q6_K (K-quants), the q4_K_M vs q4_K_S naming convention, and IQ-quants with imatrix calibration. The most-deployed quantization scheme in the world.