AI Fundamentals

What Is LoRA? Low-Rank Adaptation Explained for Fine-Tuning LLMs

LoRA fine-tunes a large language model by freezing its weights and training two small matrices per layer. Learn how it works, the parameter math, what rank means, and its limits.

Approximately 5 min read

LoRA, short for Low-Rank Adaptation, is a way to fine-tune a large language model without changing the model’s original weights. Instead of updating every parameter, you freeze the pretrained model and train a pair of small matrices that describe a change to each selected layer. The idea comes from the 2021 paper “LoRA: Low-Rank Adaptation of Large Language Models” by Edward Hu and colleagues.

If you already know what a fine-tune is, LoRA is the answer to a practical question: how do you teach a big model a new job when you cannot afford to store and train a full second copy of it?

The problem LoRA solves

Full fine-tuning updates every weight. That has two costs. Training needs memory for the weights and also for the gradients and optimizer state of every one of them. And each fine-tuned model is a complete new copy, so ten tasks means ten full-size checkpoints.

The LoRA authors framed this directly for GPT-3 175B: storing a separately fine-tuned instance per task is prohibitively expensive. Their paper reports that, compared with fully fine-tuning GPT-3 175B with Adam, LoRA cut the number of trainable parameters by about 10,000 times and the GPU memory requirement by about 3 times. Those are the authors’ reported results on that setup, not a guarantee for every model or framework.

How LoRA works

A transformer layer contains large weight matrices. Call one of them W, with shape d × k. Full fine-tuning learns an update ΔW of the same shape, so the new weight is W + ΔW.

LoRA’s central assumption is that the update needed for a new task has low “intrinsic rank”: it can be described with far fewer numbers than the full matrix holds. So rather than learning ΔW directly, LoRA writes it as the product of two thin matrices:

and learns only those. The adapted layer computes W·x + (α/r)·B·A·x, where r is the rank (a small number you choose) and α is a scaling constant. The original W stays frozen the whole time.

In the usual setup, A starts with small random values and B starts at zero. That means B·A is zero at the beginning, so training starts from exactly the pretrained model’s behaviour and moves away from it only as the small matrices learn.

The parameter math

The saving is easy to see with arithmetic. Take one illustrative square matrix with d = k = 4096. Full fine-tuning would train 4096 × 4096 = 16,777,216 values for that matrix. LoRA trains r × (d + k):

Rank r LoRA trainable values Share of the full matrix
4 32,768 about 0.20%
8 65,536 about 0.39%
16 131,072 about 0.78%
64 524,288 about 3.1%

These numbers are plain arithmetic on a hypothetical matrix size, not measurements from any specific model. Real models have many such matrices, and you choose which ones get adapters, so the whole-model share depends on that choice and on the rank.

Why inference does not get slower

A separate “adapter layer” inserted into a network adds extra computation at every forward pass. LoRA avoids that. Because the update is just B·A, you can add it into the original weight once, after training:

W' = W + (α/r)·B·A

The merged W' has the same shape as W, so a merged model runs like any ordinary checkpoint. The paper states that, unlike adapters, LoRA introduces no additional inference latency. You can also keep the small matrices separate, which lets one frozen base model serve several tasks by swapping adapter files that are tiny compared with the base.

What rank means in practice

Rank controls capacity. A low rank means the update lives in a very small subspace: cheap to train and store, but limited in how much it can change. A higher rank can express more change at a higher cost. The paper includes an empirical study of rank deficiency in language-model adaptation, motivating the low-rank assumption. A task that needs the model to learn a lot of genuinely new knowledge or behaviour may need a higher rank, or may not suit a low-rank update at all.

Rank is one of several choices you make when using LoRA:

Where LoRA falls short

LoRA is a parameter-efficient method, not a free lunch.

How LoRA fits with other pieces

LoRA sits inside the broader picture of fine-tuning a pretrained model for a new job. The matrices it freezes and the ones it trains are the same kind of object described in parameters, weights and biases. If you want the memory side of the story, the VRAM guide for local LLMs shows how weights dominate what a GPU must hold.

Key takeaways

Sources and further reading