What Is LoRA? Low-Rank Adaptation Explained for Fine-Tuning LLMs
LoRA fine-tunes a large language model by freezing its weights and training two small matrices per layer. Learn how it works, the parameter math, what rank means, and its limits.
Approximately 5 min read
LoRA, short for Low-Rank Adaptation, is a way to fine-tune a large language model without changing the model’s original weights. Instead of updating every parameter, you freeze the pretrained model and train a pair of small matrices that describe a change to each selected layer. The idea comes from the 2021 paper “LoRA: Low-Rank Adaptation of Large Language Models” by Edward Hu and colleagues.
If you already know what a fine-tune is, LoRA is the answer to a practical question: how do you teach a big model a new job when you cannot afford to store and train a full second copy of it?
The problem LoRA solves
Full fine-tuning updates every weight. That has two costs. Training needs memory for the weights and also for the gradients and optimizer state of every one of them. And each fine-tuned model is a complete new copy, so ten tasks means ten full-size checkpoints.
The LoRA authors framed this directly for GPT-3 175B: storing a separately fine-tuned instance per task is prohibitively expensive. Their paper reports that, compared with fully fine-tuning GPT-3 175B with Adam, LoRA cut the number of trainable parameters by about 10,000 times and the GPU memory requirement by about 3 times. Those are the authors’ reported results on that setup, not a guarantee for every model or framework.
How LoRA works
A transformer layer contains large weight matrices. Call one of them W, with shape d × k. Full fine-tuning learns an update ΔW of the same shape, so the new weight is W + ΔW.
LoRA’s central assumption is that the update needed for a new task has low “intrinsic rank”: it can be described with far fewer numbers than the full matrix holds. So rather than learning ΔW directly, LoRA writes it as the product of two thin matrices:
B, with shaped × rA, with shaper × k
and learns only those. The adapted layer computes W·x + (α/r)·B·A·x, where r is the rank (a small number you choose) and α is a scaling constant. The original W stays frozen the whole time.
In the usual setup, A starts with small random values and B starts at zero. That means B·A is zero at the beginning, so training starts from exactly the pretrained model’s behaviour and moves away from it only as the small matrices learn.
The parameter math
The saving is easy to see with arithmetic. Take one illustrative square matrix with d = k = 4096. Full fine-tuning would train 4096 × 4096 = 16,777,216 values for that matrix. LoRA trains r × (d + k):
Rank r |
LoRA trainable values | Share of the full matrix |
|---|---|---|
| 4 | 32,768 | about 0.20% |
| 8 | 65,536 | about 0.39% |
| 16 | 131,072 | about 0.78% |
| 64 | 524,288 | about 3.1% |
These numbers are plain arithmetic on a hypothetical matrix size, not measurements from any specific model. Real models have many such matrices, and you choose which ones get adapters, so the whole-model share depends on that choice and on the rank.
Why inference does not get slower
A separate “adapter layer” inserted into a network adds extra computation at every forward pass. LoRA avoids that. Because the update is just B·A, you can add it into the original weight once, after training:
W' = W + (α/r)·B·A
The merged W' has the same shape as W, so a merged model runs like any ordinary checkpoint. The paper states that, unlike adapters, LoRA introduces no additional inference latency. You can also keep the small matrices separate, which lets one frozen base model serve several tasks by swapping adapter files that are tiny compared with the base.
What rank means in practice
Rank controls capacity. A low rank means the update lives in a very small subspace: cheap to train and store, but limited in how much it can change. A higher rank can express more change at a higher cost. The paper includes an empirical study of rank deficiency in language-model adaptation, motivating the low-rank assumption. A task that needs the model to learn a lot of genuinely new knowledge or behaviour may need a higher rank, or may not suit a low-rank update at all.
Rank is one of several choices you make when using LoRA:
- Which weight matrices receive adapters.
- The rank
rand the scalingα. - The learning rate and number of training steps, which still matter as much as in any fine-tune.
Where LoRA falls short
LoRA is a parameter-efficient method, not a free lunch.
- Memory is reduced, not removed. The frozen base model still has to sit in memory during training, and activations still cost memory. The big saving is on gradients and optimizer state for the trainable parameters.
- Quality is task-dependent. The paper reports LoRA matching or beating full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3 on the tasks it tested. That does not mean it matches full fine-tuning on every task, dataset or model.
- It does not shrink the base model. LoRA changes how you train, not how big the deployed model is. If model size is the constraint, that is a job for quantization rather than adapters.
How LoRA fits with other pieces
LoRA sits inside the broader picture of fine-tuning a pretrained model for a new job. The matrices it freezes and the ones it trains are the same kind of object described in parameters, weights and biases. If you want the memory side of the story, the VRAM guide for local LLMs shows how weights dominate what a GPU must hold.
Key takeaways
- LoRA freezes the pretrained weights and trains two small matrices whose product is the weight update.
- Trainable parameters scale with
r × (d + k)instead ofd × k, which is why small ranks are so cheap. - After training, the update can be merged into the original weight, so inference latency does not increase.
- Rank is a capacity setting; low rank is efficient but not unlimited.
- Reported savings and quality results come from the paper’s own experiments, and your results will depend on your model, data and settings.