LoRA: Fine-Tune a Huge Model Without Copying Its Weights

In 2021, the way to fine-tune a large language model changed direction. Instead of copying every weight and retraining it, the weights are frozen and only a small insert gets trained. Two years later, those weights were compressed into 4-bit quantization. Together they make a 65-billion-parameter model trainable on a single 48 GB GPU. That is a mechanism explainer rather than a release report, and the shift is felt less on the bill than on who gets to experiment.

2021: Freeze the Original Weights, Train a Small Insert

LoRA stands for Low-Rank Adaptation of Large Language Models. That is the title of a paper by Edward J. Hu with Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, and five other co-authors. The first version appeared on arXiv in June 2021 and at ICLR 2022.

The idea fits in one sentence: freeze the old weights, then add a trainable matrix delta to each layer. That delta is never stored in full, because it is computed from two much thinner matrices. The change to the weight can still be written as the product of those two small matrices.

Training is designed so the model does not jump at the start. One matrix is randomly initialised and the other starts at zero, so the delta begins at exactly zero. Its output is then scaled by alpha over r, which keeps hyper-parameters from being retuned every time r changes.

Picture the model weights as a freshly painted concrete wall. Before, to change even a little, you had to demolish that wall. LoRA glues thin panels on top, and those panels come off when you switch tasks.

On GPT-3 175B, trainable parameters drop by ten thousand and GPU memory drops threefold. The paper also budgets 18 million parameters, roughly 35 MB in FP16, across all 96 layers. That equals rank 8 for one attention weight type, or rank 4 for two. After training, the insert is added into the original weights, so the model runs as before with no extra serving cost.

Why a Low-Rank Matrix Is Enough

Low rank here is not empty jargon. It means the matrix carries only a few strong directions, while the rest are close to zero. The surprising finding from 2021: the change the model actually needs has a very small order.

The paper tests rank from 1 to 64 on two datasets, adapting two attention matrices. On WikiSQL, rank 1 already reaches 73.4 percent, and rank 8 edges up to 73.8. On MultiNLI, the number moves from 91.3 at rank 1 to 91.6 at rank 8. Pushing on to rank 64 adds nothing.

These numbers come from Table 6 of the LoRA paper, measured on GPT-3 175B. So rank is not a dial to always turn up. Too small and the adapter lacks capacity for a demanding task. Too large and memory plus overfitting risk rise with little gain. The range people reach for most sits between 8 and 64, which is a practical habit rather than a number the paper sets.

The Bottleneck That Arrived Two Years Earlier

The road to LoRA did not start in 2021. In 2019, Neil Houlsby and his team published Parameter-Efficient Transfer Learning for NLP at ICML, inserting a small module twice in every Transformer layer.

The module works like a bottle with a narrow neck: it enters through a small dimension and widens back out. The parameter count per layer is only 2md plus d plus m, with m far smaller than d, which in practice runs about 0.5 to 8 percent of the original model size. On GLUE it lands within 0.8 percent of full fine-tuning while using just 3.6 percent of parameters per task.

The problem sits on the inference side: every layer takes two extra matrix multiplications. LoRA promises something stronger, because its insert merges into the weights and running the model costs no extra. The umbrella term for all of this is parameter-efficient fine-tuning.

QLoRA in 2023: Four-Bit Weights That Still Train

In 2023, Tim Dettmers and his team dropped one more level. The QLoRA paper stores the base weights in 4-bit while computation still runs at bfloat16 precision.

The trick sits in three small details. NF4 is a 4-bit data type tuned for normally distributed weights. Double quantization quantizes the quantization constants again, saving about 0.37 bits per parameter, roughly 3 GB on a 65-billion-parameter model. Paged optimizers move memory spikes into NVIDIA unified memory.

The result falls from more than 780 GB to less than 48 GB, which fits on a single GPU. That is the paper's own figure, not somebody else's estimate. It is often retold as a 70-billion model, while the number actually tested is 65 billion, and that gap matters because a wrong figure becomes an excuse to call something too expensive.

Memory saved does not cost quality here. On MMLU, NF4 with double quantization reaches 53.1 while the bfloat16 baseline reaches 53.0, and FP4 trails by about one point. The Guanaco family was trained across more than a thousand models, and its smallest member, 7 billion parameters, needs only 5 GB to serve.

Why Full Fine-Tuning Is a Memory Hog

Full fine-tuning changes every weight, so every weight needs room, gradients, and optimizer state. Adam keeps momentum and variance, two extra copies for every trained parameter. The rough arithmetic: 65 billion parameters in FP16 alone is 130 GB. The QLoRA paper puts average finetuning needs for a model that size above 780 GB. The gap is clear enough: optimizer state is far larger than the weights.

Picture a warehouse that stores goods and then rents a second warehouse just for the inventory list. LoRA cuts that side, because optimizer state is only built for the few parameters it trains. The LoRA paper records that fully finetuning models of 11 billion parameters and up needs more than one large GPU server.

Fine-Tuning Is Not RAG

Plenty of people assume fine-tuning and RAG do the same thing. Their goals differ: fine-tuning changes behaviour, RAG adds facts.

Fine-tuning helps with tone, output format, house terminology, and format compliance. RAG helps with recent figures, internal documents, and anything that changes after training.

Picture a cook in a kitchen. Fine-tuning retrains the cook on procedure: how to greet, how to cut, how to plate. RAG hands the cook a fresh menu at every shift. The difference that gets mentioned least is cost: RAG can be dropped tomorrow, fine-tuning cannot be peeled off that easily.

The Danger of Teaching Facts the Wrong Way

If a fact is trained on, it becomes part of the weights, and weights have no expiry date. Data that changes next year stays stuck in there. A document pulled at answer time is always the latest version, because the source is being read right then.

A model trained heavily to recall tends to emit the pattern it memorised faster, while retrieval from an outside source always has a verification step in the middle. Memorisation has no such step.

So hallucination is not purely a matter of a dumb model. Sometimes it is a bad training choice, because we taught the model to remember rather than to check. Take a concrete case: a customer service number changes now and then, and it is always easier to copy that number into an external document than to make a model memorise it. People still ask the model first, because the answer arrives faster. That gap between the easy path and the correct path is where the failure comes from, and no rank setting closes it.

Deeper Dive: Who Gets to Experiment Now

Before 2021, trying to adapt a model meant bringing your own machine. After it, the bar drops to one second-hand GPU or a single 24 GB card on a desk, which changes who is allowed to run experiments.

Four things people usually skip when discussing LoRA:

• The LoRA matrices can be merged into the base weights, so serving the model costs nothing extra.
• One base model can serve many adapters at once, each one dedicated to a single task.
• A trained adapter is small, which makes it easy to share and re-run.
• Learning is expensive when one experiment is expensive, and cheap experiments get retried far more often.

That fourth point is the decisive one. Knowledge in this field moves partly because too few experiments fail cheaply. If one iteration eats a week, people typically try twice and give up. If one iteration eats a night, people try ten times. That shift matters more than the memory saving.

Fine-tuning is not cheap, though, and data quality still decides the outcome. A cheap second-hand GPU does not excuse a messy example set. As of 2026, this route is the default path rather than a research-lab trick.

Numbers You Should Read Carefully

One licensing note often gets skipped. Open-weight models arrive under varied licences, from fully commercial ones to ones that restrict redistribution. The licence follows the base model, so downloading someone else's adapter does not clear you of it.

Claims about how many GPUs a particular model needs get repeated without sources. The figures here come from the two original papers. Market GPU prices move fast, so they are deliberately left out, and parameter counts for circulating open-weight models differ between versions too.

One question is left open. If adaptation only needs a small number of parameters, is copying all the weights still worth doing? In 2021 the answer was clear: most of those weights never change. That answer still holds as of 2026, and no paper has yet overturned it.