Fine-tuning, LoRA, and PEFT

How to adapt a pretrained model to your task without retraining all 70B parameters. Full fine-tuning vs feature-based extraction vs parameter-efficient methods (LoRA, QLoRA, adapters). The LoRA math — why a rank-8 update matrix captures most of the adaptation signal — falls out of the intrinsic-dimensionality argument (Aghajanyan 2020).

§1 Three fine-tuning approaches — full, feature, PEFT §2 LoRA — the math and the intrinsic-dim argument §3 The PEFT family — adapters, prefix, IA³, prompt tuning

§1 Three fine-tuning approaches — full, feature, PEFT
Three distinct families of adapting a pretrained model: (1) full fine-tuning — update every parameter; (2) feature extraction — freeze the backbone, train a new head; (3) parameter-efficient — keep the backbone frozen, train small adapters. Each is right for different memory budgets, task domains, and deployment models. This section walks the trade-offs and the cases where each one wins.
§2 LoRA — the math and the intrinsic-dim argument
LoRA (Hu 2021) replaces a fine-tuning update ΔW with a rank-r decomposition ΔW = B · A, where B ∈ ℝ^{d×r}, A ∈ ℝ^{r×d}, and r ≪ d. Only A and B train; the base W is frozen. Trainable params drop by ~d/r — typically 100-1000×. Why does rank 8-16 capture most of full-FT learning on a 7B+ model? The empirical justification is Aghajanyan 2020 "Intrinsic Dimensionality" — fine-tuning lives on a ~200-1000-dim manifold of parameter space. LoRA exploits this directly.
§3 The PEFT family — adapters, prefix, IA³, prompt tuning
The PEFT space is much bigger than just LoRA. Adapters (Houlsby 2019) insert small bottleneck MLPs between transformer blocks. Prefix tuning (Li 2021) prepends trainable soft tokens to the input. Prompt tuning (Lester 2021) is a simpler variant. IA³ (Liu 2022) scales activations by learned per-feature multipliers. Each addresses a different aspect of what should we adapt to change the model behavior. LoRA + QLoRA won the production race, but each variant has its niche.

← ALL CHAPTERS