Every transformer wraps each attention block and FFN block with a normalisation operation. Without it, training is unstable: activations drift to large magnitudes layer by layer; the softmax in attention saturates; gradients vanish or explode. Ba 2016 “Layer Normalization” introduced LayerNorm as the fix that finally worked for sequence models. Unlike BatchNorm (Ioffe & Szegedy 2015) which normalises across the batch dimension, LayerNorm normalises across the feature dimension of each token independently. The operational consequence: LayerNorm works at batch size 1, works during inference the same way it works during training, works with variable-length sequences. These constraints are why every transformer uses it.

The formula

For a single token’s activation vector x ∈ ℝ^d (one row of the activations matrix), LayerNorm computes:

Two-stage operation: normalise (parameter-free, per-token), then affine (learned, per-feature). The normalise stage is what stabilises the activations; the affine stage is what gives the layer expressivity back — without γ, β the network would lose the ability to represent any non-unit-variance, non-zero-mean intermediate distribution.

LayerNorm normalisation A per-token normalisation: subtract the mean across feature dimensions, divide by the standard deviation (across feature dimensions), then apply a learned per-feature affine transform γ·x + β. Operates within a single token's activation vector — independent of batch size, independent of other tokens. Ba 2016. The default normalisation in original transformers (post-norm position). ’s scope is one token’s activation vector. The statistics μ and σ² come from that one vector’s d feature values; they have nothing to do with other tokens in the same batch or sequence.

Why per-feature γ and β

Both γ and β are vectors of length d (one entry per feature dimension), and they’re learned. Why?

Without the affine stage, every LayerNorm output has zero mean and unit variance per token — that’s a hard structural constraint. But the network might need certain features to be larger than others (e.g., positional features dominating semantic ones, or one head’s contribution being weighted differently), or to have non-zero mean (e.g., the residual stream encoding a sentence-level bias). The γ, β stage gives the network back this freedom: it can choose to keep the unit-variance constraint (set γ = 1, β = 0) or scale specific features up/down (γ_j ≠ 1) or shift them (β_j ≠ 0).

The expressivity argument: without γ, β, two LayerNorms with the same input always produce the same output regardless of the surrounding network. With γ, β, the layer can adapt its output distribution to what the next layer needs. In practice γ tends to learn values in [0.1, 10] and β in [−0.5, 0.5] — small adjustments to the parameter-free normalisation, not radical changes.

LayerNorm vs BatchNorm — which axis are we normalising?

Now make it run

The C kernel implements LayerNorm and verifies the output has zero mean and unit variance per row when γ = 1 and β = 0:

static void layernorm(const float* X, const float* g, const float* b,
                      float* Y, int N, int D)
{
    const float eps = 1e-5f;
    for (int i = 0; i < N; i++) {
        const float* x = X + (size_t)i * D;
        float* y = Y + (size_t)i * D;
        /* mean */
        float mu = 0;
        for (int j = 0; j < D; j++) mu += x[j];
        mu /= D;
        /* variance */
        float var = 0;
        for (int j = 0; j < D; j++) { float dx = x[j] - mu; var += dx * dx; }
        var /= D;
        float inv = 1.0f / sqrtf(var + eps);
        /* normalise + affine */
        for (int j = 0; j < D; j++) y[j] = g[j] * ((x[j] - mu) * inv) + b[j];
    }
}

Output:

LayerNorm  output (γ=1, β=0):  max |row mean|     = 1.44e-06   (should be ~0)
LayerNorm  output (γ=1, β=0):  max |row var − 1|  = 3.28e-06   (should be ~0)

LayerNorm produces zero-mean, unit-variance rows to roundoff. The 1e-6 errors are float32 epsilon — accumulating d = 512 sums in float32 produces this much drift. Modern transformer implementations sometimes compute the LayerNorm statistics in fp32 even when the rest of the layer is in fp16/bf16, precisely to keep this error from compounding.

The gradient — why this helps training

The gradient through LayerNorm is more interesting than it looks. Working out ∂L/∂x from ∂L/∂y:

The two correction terms are what makes LayerNorm a gradient whitener LayerNorm effect LayerNorm's per-token statistics introduce coupling between each x_j and every other feature in the same row (through μ and σ²). The resulting gradient ∂L/∂x_j subtracts both a mean and a variance projection from the naive gradient, which empirically 'whitens' the loss landscape — gradients across features become less correlated, the loss landscape less curved in any specific direction. This is what allows deep networks (12+ layers) to train without diverging. . The mean correction removes the component of the upstream gradient that affects only the mean (the network can’t change the input mean — LayerNorm just subtracts it back out). The variance correction removes the component that affects only the variance. What’s left is the gradient signal that actually changes the direction of x.

The empirical effect: without LayerNorm, transformers above ~6 layers diverge during training — activations explode at intermediate layers. With LayerNorm, the same architecture trains stably at 96+ layers. The “gradient whitening” is what enables the depth that makes large transformers work.

Next: §14.2 — RMSNorm and pre-norm placement. The “drop the mean” simplification that every modern open model uses, and the placement choice (pre-norm before the sublayer vs post-norm after) that affects gradient flow.