”Training a neural network” sounds elaborate. It isn’t. It’s finding the model parameters that minimise a loss function averaged over training examples. That’s the whole formal procedure — the rest is implementation. The loss function says which mistakes you care about; gradient descent (next section) says how to find the parameters that minimise the average. Get the loss wrong and the model optimises for the wrong thing — perfect MSE on outlier-contaminated data still gives a regression line that lies; cross-entropy mismatched to softmax outputs underflows; rank-aware ML systems whose loss isn’t rank-aware miss the point of ranking. The loss function is your contract with the model. This section nails down the standard losses and the empirical-risk-minimisation framework, both refreshed for an audience that learned this material when “regularisation” still mostly meant “ridge regression.”

A loss is a number per example

A loss function core concept A function L(prediction, truth) → ℝ⁺ that quantifies how wrong a prediction is. Examples: MSE, MAE, cross-entropy, hinge, focal. Choice of loss determines what the model optimises for — picking the loss is as important as picking the architecture. takes one prediction and one ground-truth label and returns a non-negative number: how wrong was that prediction? Zero is perfect; bigger is worse. The empirical risk core concept The average loss over a training set: R̂(θ) = (1/N) Σ L(f_θ(xᵢ), yᵢ). Training a model is empirical risk minimisation: find θ that minimises R̂. The 'risk' framing is from statistical decision theory (Vapnik et al.) and made it back into ML via the EBM (empirical risk minimisation) framework of Vapnik & Chervonenkis 1971. — what you actually minimise during training — is the average loss across your training set:

“Training” is: pick a θ that minimises R̂(θ). That’s it. The rest of this chapter — gradient descent, SGD, Adam — is how you find that minimising θ. Everything in deep learning since 2012 falls under this framework; the field is sometimes literally called empirical risk minimisation framework The framework that defines machine learning as picking parameters θ to minimise the empirical risk R̂(θ) = (1/N) Σ L(f_θ(xᵢ), yᵢ). Introduced by Vapnik & Chervonenkis 1971; the foundational principle that lets us 'train' models — without it there's no formal definition of what training is supposed to do. (ERM) when discussed at the theory level.

The two big losses

MSE loss · regression Mean Squared Error: L(ŷ, y) = (ŷ − y)². Smooth, differentiable everywhere, gradient grows linearly with residual. The default loss for regression. Implicitly assumes Gaussian noise on the targets (it's the negative log-likelihood under that model). is the default for regression. Three reasons it’s the canonical choice:

• Smooth and differentiable. Easy to optimise with gradient methods.
• Implicit Gaussian assumption. If you assume your residuals are Gaussian, MSE is exactly the negative log-likelihood — minimising MSE = maximising likelihood under a Gaussian noise model. The CLT (Ch.5 §2) is the reason this assumption is roughly right surprisingly often.
• Strict convexity in the prediction. No local minima around the prediction; the loss landscape is well-behaved with respect to the output.

The downside: MSE squares the residual, so outliers dominate. One bad data point with residual 10 contributes 100 to the loss; ten typical points with residual 1 each contribute only 10 combined. MAE loss · regression · robust Mean Absolute Error: L(ŷ, y) = |ŷ − y|. Gradient magnitude is constant (sign(ŷ − y)) — robust to outliers but non-smooth at zero residual. Closed-form minimiser is the conditional median. (mean absolute error) fixes this — its gradient magnitude is constant, so outliers don’t dominate — at the cost of being non-smooth at zero residual.

Huber loss loss · regression · robust A piecewise function that's MSE near zero residual (smooth, fast convergence) and MAE in the tails (robust to outliers): L(r) = r²/2 if |r| < δ, else δ(|r| − δ/2). Default in many robust regression libraries. is the practical compromise — quadratic near zero (smooth, fast convergence) and linear in the tails (robust). Default in many robust-regression libraries.

Drag w; flip between MSE, MAE, Huber. Notice the optimum moves — different losses optimise to different parameters on the same data. The dataset has a few outliers (the cluster of points well off the trend line); MSE pulls toward them, MAE/Huber don’t.

Cross-entropy — the loss for classification

For a probabilistic classifier producing softmax outputs ŷ = softmax(z) over C classes (where z are the pre-softmax logits), the standard loss is cross-entropy loss · classification L(ŷ, y) = −Σ_c y_c log ŷ_c, where y is a one-hot encoding of the true class and ŷ is a probability distribution (softmax output). Equivalent to the negative log-likelihood of the true class under the model's distribution. The unique loss that, when minimised against a correctly-specified probabilistic model, recovers the true conditional distribution. :

Three things to know about cross-entropy:

• It’s negative log-likelihood. Minimising cross-entropy = maximising the likelihood the model assigns to the training data. Same identity as MSE-vs-Gaussian-likelihood, just for categorical distributions instead of Gaussian.
• Pairs naturally with softmax. The gradient of cross-entropy with respect to the pre-softmax logits is a clean ŷ − y (one-hot subtracted from probabilities) — easy to backprop. Implementations often fuse softmax and cross-entropy into one numerically-stable op (log_softmax + nll) to avoid overflow at large logits.
• It heavily penalises confident wrong predictions. If the model assigns 0.001 to the true class, the loss is −\log 0.001 ≈ 6.9. If it assigns 0.99, the loss is 0.01. The relationship between probability and loss is logarithmic, so being wrong-and-confident is severely punished.

This makes cross-entropy the right loss for probability calibration — models trained on it produce probability estimates that are roughly correctly scaled, not just discriminative. (Guo et al. 2017, “On Calibration of Modern Neural Networks,” ICML — note that this is a regression: large NNs are typically over-confident even when trained on CE; calibration techniques like temperature scaling fix this post-hoc.)

Why minimise empirical risk?

You’re minimising the average loss on your training data. What you actually care about is the loss on future data — the true risk theory concept R(θ) = expected value of L(f_θ(x), y) under (x,y) sampled from the true data distribution D. We cannot compute it directly (D is unknown), so we minimise the empirical proxy R̂(θ). Generalisation theory bounds the gap between R̂ and R. R(θ) = E[L(f_θ(x), y)] over the actual data distribution. The empirical risk is just the Monte-Carlo estimate of the true risk based on the training set you have.

By the √N law from Ch.5 §1, the empirical risk estimates the true risk with standard error σ_L/√N. So if your training set is small, your empirical risk is a noisy proxy for what you actually care about — and the parameter that minimises empirical risk is generally not the parameter that minimises true risk. The gap is generalisation error theory concept The gap between true risk R(θ) and empirical risk R̂(θ) at a particular θ. Small generalisation error = the model performs on new data roughly as well as it does on training data. Large gap = overfitting. Controlled by model capacity, training set size, regularisation, and (modern view) implicit biases of the optimiser. Then → now: the bias-variance / VC-dimension theory you learned in 2007 was the dominant framework — capacity-controlled bounds. Modern ML (especially deep learning) has weakened this framework: huge over-parameterised networks generalise WELL despite VC dimension predicting otherwise. The modern explanations involve implicit biases of SGD (Ch.8 §2-3), feature learning, and effective model capacity rather than nominal parameter count. .

This is why every training loop pairs the empirical-risk minimisation with techniques to keep the parameters from overfitting:

• A held-out validation set. Compute the loss on data the model wasn’t trained on; stop training when validation loss stops improving.
• Weight decay / L2 regularisation. Penalise large parameter values, biasing toward simpler solutions.
• Early stopping, dropout, data augmentation, batch normalisation. Each reduces effective capacity in a different way.

We’ll see how each interacts with the optimiser in §8.3.

Some other losses worth knowing

A non-exhaustive but useful list:

Modern ML pipelines often mix several losses with weighted sums — e.g., RLHF combines a policy loss with a value loss with a KL regulariser to the SFT model. The choice of which losses to combine, and with what weights, is the dominant lever for shaping behaviour.

END OF CH.8 §1 — Loss functions & empirical risk.
Built: LossLandscape viz (slide w, see how the optimum moves depending on whether you optimise MSE / MAE / Huber — different losses give different fits on the same data). Three recall items: easy (loss as contract), medium (MSE vs MAE and the implicit noise assumptions), hard (the empirical-vs-true-risk gap derived from the √N law).
Coming next: §8.2 — Gradient descent and stochastic gradient descent. How to actually find the θ that minimises empirical risk.