Suppose you have $10M of compute and want to train the best possible LLM. Should you train a 200B model on 200B tokens, or a 30B model on 1.5T tokens? Both use roughly the same compute. The answer was unclear until 2022, when DeepMind’s Chinchilla paper (Hoffmann 2022) ran 400+ training runs at varying (N, D) and fit a clean empirical law. The result: N and D should grow TOGETHER, and for compute-optimal training, D ≈ 20 · N. This invalidated the prior Kaplan 2020 prescription (“more params, fewer tokens”) and reshaped every subsequent model release. This section derives the scaling law, runs a kernel that fits it to synthetic data and solves the compute-optimal frontier, and explains why modern releases (Llama 3, Qwen 3) intentionally over-train past Chinchilla optimum.

The scaling law

The empirical Chinchilla fit is:

The scaling law empirical fit The empirical formula L(N, D) = E + A/N^α + B/D^β fit by Hoffmann 2022 on a sweep of model training runs. E is the irreducible loss (entropy of text, ~1.69 nats per token); A/N^α captures model-size limitation; B/D^β captures data-size limitation. The exponents α≈0.34 and β≈0.28 are small (sublinear returns) and close to each other (symmetric scaling in N and D). The constraint of fixed compute C = 6·N·D then yields the compute-optimal D/N ≈ 20. is the single most-cited empirical result in modern ML. It says: both model size and dataset size are diminishing-returns resources, and they should be increased together. Hoarding one while ignoring the other is wasteful.

Compute-optimal frontier

For a fixed compute budget C, what’s the best (N, D)?

The result is striking. The compute-optimal D/N ratio is approximately constant across budget sizes — it depends only on the scaling-law parameters, not on how much compute you have. With Chinchilla’s α=0.34 and β=0.28, that ratio is ~20×: optimal training feeds 20 tokens per parameter.

Now make it run

The kernel below generates synthetic (N, D, L) data from the empirical law plus noise, fits the law’s three parameters by gradient descent on squared log-residual, and computes the compute-optimal frontier across 8 budgets from 10¹⁸ to 10²⁵ FLOPs:

    int steps = 200000;
    for (int s = 0; s < steps; s++) {
        double gE = 0, gA = 0, gAl = 0, gB = 0, gBe = 0;
        for (int i = 0; i < n; i++) {
            double Nv = N_arr[i], Dv = D_arr[i], Lt = L_arr[i];
            double Na = pow(Nv, p->alpha), Db = pow(Dv, p->beta);
            double Lp = p->E + p->A / Na + p->B / Db;
            double r = Lp - Lt;
            gE  += r;
            gA  += r / Na;
            gAl += r * (-p->A) / Na * log(Nv);
            gB  += r / Db;
            gBe += r * (-p->B) / Db * log(Dv);
        }
        p->E     -= lr      * gE  / n;
        p->A     -= lr * 50 * gA  / n;

Output:

Compute-optimal frontier  (FLOPs C = 6 · N · D)
    C (FLOPs)       N*           D*          D*/N*    L(N*,D*)
    1e+18         1.20e+08    1.38e+09    11.5     3.393
    1e+19         3.52e+08    4.73e+09    13.4     2.940
    1e+20         1.03e+09    1.62e+10    15.7     2.611
    1e+21         3.01e+09    5.53e+10    18.3     2.371
    1e+22         8.40e+09    1.98e+11    23.6     2.197
    1e+23         2.46e+10    6.79e+11    27.6     2.070
    1e+24         7.19e+10    2.32e+12    32.3     1.978
    1e+25         2.10e+11    7.93e+12    37.7     1.911

The D*/N* ratio ranges from ~12 (small models) to ~38 (large models), centred on the empirical “~20×” Chinchilla number. The deviation from a strictly-constant ratio comes from the slight asymmetry in α vs β (small differences in scaling exponents push the optimum slightly in the larger model’s direction at higher budgets).

Why GPT-3 was under-trained

The difference was a methodological subtlety. Kaplan’s experiments used fixed learning-rate schedules at all scales. Hoffmann’s used carefully-tuned learning-rate schedules calibrated per (N, D) pair. The tuned schedules revealed that the data-axis exponent β was much higher than Kaplan’s untuned fit suggested. The “correct” compute-optimal recipe shifted from “parameters dominate” to “balanced N and D.”

Llama 2 (2023) was the first major release explicitly following Chinchilla: 7B, 13B, 34B, 70B all trained on ~2T tokens. Llama 3 (2024) went further — 8B and 70B trained on ~15T tokens, well PAST Chinchilla optimum.

Why modern training intentionally over-trains

If Chinchilla is right, why train past D ≈ 20·N? Because inference cost matters too.

Over-training modern recipe Training a smaller-than-Chinchilla-optimal model on MORE-than-Chinchilla-optimal tokens. Wastes training compute (the marginal loss reduction from extra tokens diminishes) but saves inference compute throughout the model's lifetime (the smaller model has 2-10× lower per-token serving cost). Llama 2/3, Mistral, Qwen, DeepSeek all over-train aggressively. The training cost is paid once; the inference savings compound forever. is the dominant recipe for modern deployed models. The “Chinchilla optimum” is what you’d choose if compute was the only resource. In the real world, inference cost is a much larger budget item, so you trade extra training compute for a smaller deployed model.

END OF CH.16 — Pretraining.
§1 (the loop: NTL, mixed precision, gradient accumulation, all-reduce) · §2 (the data pipeline: Common Crawl → FineWeb, MinHash dedup, why quality matters) · §3 (Chinchilla scaling laws + the over-training rationale).

Next: Ch.17 — Mixture of Experts. Sparse activation, why a 47B-param model can be 13B compute, the load-balancing problem.