Interactive Explainer
Full fine-tuning of a 70B model touches 70 billion parameters. LoRA touches 200 million. Quality is often indistinguishable, and the checkpoint is 300× smaller. Slide the rank and see the numbers move; then watch a live SVD recover a target matrix as $r$ grows.
The change you want to apply to a pretrained weight is typically low-rank (Hu et al., 2021). You don't need 70B parameters of update budget — you need a tiny correction. So instead of fine-tuning $W \in \mathbb{R}^{d \times d}$, parameterise the update as
where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times d}$ with $r \ll d$. The base $W_0$ stays frozen; only $A$ and $B$ get gradients. The scalar $\alpha/r$ keeps the effective step size stable across ranks. At initialisation $B = 0$, so the LoRA term is identically zero and the model is exactly the pretrained one — a guaranteed starting point.
The Hu et al. paper measured the intrinsic dimensionality of fine-tuning a pretrained transformer on downstream tasks and found it was tiny — often under $r = 8$ for $d = 4096$. Two ways to think about this:
Below: a synthetic "true update" $\Delta W^\star \in \mathbb{R}^{32 \times 32}$ with a heavy-tailed singular-value spectrum (typical of real fine-tuning). The rank-$r$ truncated SVD is the best rank-$r$ approximation by Eckart–Young. Slide $r$ and watch the approximation error and the captured spectrum.
Left: target $\Delta W^\star$ (top) and its rank-$r$ approximation (bottom). Right: cumulative spectral energy captured by the top-$r$ singular values. Heavy-tailed spectrum needs very small $r$ for > 95% energy.
| Method | Trainable params | Ratio vs full | Adapter on disk |
|---|---|---|---|
| Full fine-tune | – | 1.00× | – |
| LoRA · r=8 | – | – | – |
| QLoRA (4-bit base + r=8) | – | – | – |
At $r = 8$ for a 7B model, you have ~4M trainable parameters. You can fine-tune on a single 24 GB consumer GPU, save the adapter as a ~16 MB file, and ship it next to the base weights. The base never changes — you can layer many adapters on top.
For most LLM tasks $r \in \{4, 8, 16\}$ is enough. Larger $r$ doesn't help unless your task adapts the model significantly — e.g. injecting a brand-new modality. Sweep $r$ logarithmically if you suspect higher rank is needed.
$\alpha$ is a scalar applied to $BA$ at inference. The "$\alpha/r$" convention keeps the effective learning rate of the LoRA term roughly constant as you sweep $r$ — handy for hyperparameter search. Typical: $\alpha = 2r$ or $\alpha = 16$ regardless of $r$.
The original paper recommends $W_q$ and $W_v$ only. Newer empirical surveys (Dettmers, Pagnoni et al., 2023) often inject into all of $\{W_q, W_k, W_v, W_o\}$ plus the MLP up-and-down projections for the best quality. Cost is linear in number of injected modules.
A small dropout (0.05–0.10) inside the LoRA path stabilises training on small fine-tuning sets without affecting the base. Skip it for tiny adapters.
| Method | Trainable % | Inference overhead | Idea |
|---|---|---|---|
| Full fine-tune | 100% | 0 | train everything |
| BitFit | ~0.1% | 0 | train only biases |
| Prefix / prompt tuning | ~0.01% | +sequence length | prepend learnable tokens |
| Adapters (Houlsby '19) | ~3% | +small MLP per layer | insert bottleneck modules |
| LoRA | ~0.05% | 0 after merge | low-rank weight delta |
| QLoRA | ~0.05% | 0 after merge | LoRA on top of 4-bit base |
| DoRA (2024) | ~0.1% | 0 after merge | decompose weight into magnitude + direction; LoRA on direction |
| VeRA (2023) | ~0.01% | 0 after merge | share frozen random $A, B$ across layers; train tiny scaling vectors |
| AdaLoRA | ~0.05% | 0 after merge | allocate rank budget adaptively per layer |
The forward pass becomes
At initialisation: $A \sim \mathcal{N}(0, \sigma^2)$ (Kaiming-style), $B = 0$. So $BA = 0$ and the model output is exactly the pretrained one. Crucially, this means LoRA never damages the base before training has done anything useful.
After training, you have a choice:
QLoRA (Dettmers et al., 2023) is "LoRA on a 4-bit base." Three pieces:
The base sits frozen in 4-bit, dequantised on the fly for matmul; the LoRA path stays in fp16/bf16. End result: same accuracy as fp16 LoRA at a fraction of the memory.
from peft import LoraConfig, get_peft_model
lora_cfg = LoraConfig(
r=8, # rank
lora_alpha=16, # scale
target_modules=["q_proj", "v_proj"], # where to inject
lora_dropout=0.05,
bias="none",
)
model = get_peft_model(base_model, lora_cfg)
model.print_trainable_parameters()
# > trainable params: 4.2M all params: 7B trainable%: 0.06%
LoRA is the rare technique where the "lazy" version actually matches the careful one. For 99% of fine-tuning tasks on pretrained models > 1B params, start with LoRA $r=8$, $\alpha = 16$, $\{W_q, W_v\}$ injection. Move to QLoRA if you can't afford fp16 base. Reach for DoRA / AdaLoRA only after you've measured a real gap.