Skip to content
ML to LLMs
Labs / LLM and RAG Lab

LoRA and fine-tuning memory

Pick a model size, rank, and precision; compare trainable parameters and GPU memory for full fine-tuning, LoRA, and QLoRA.

Theory and worked examples: Fine-tuning in practice

LoRA fine-tuning memory

Synthetic Llama-like model shapes. Estimates of training state, not measurements: real jobs add framework overhead and fragmentation.

Training memory by method0255075100training memory (GB, estimate)Full fine-tuneweights: 13.48 GBgradients: 13.48 GBoptimizer states: 80.86 GBactivations: 0.82 GB108.6 GBLoRA (bf16 base)weights: 13.56 GBgradients: 0.08 GBoptimizer states: 0.48 GBactivations: 0.82 GB14.9 GBQLoRA (NF4 base)weights: 3.95 GBgradients: 0.08 GBoptimizer states: 0.48 GBactivations: 0.82 GB5.3 GB24 GB GPU
weightsgradientsoptimizer statesactivations

W=W0+αrBAW = W_0 + \tfrac{\alpha}{r}BA: rank 16 trains r(din+dout)r(d_{in}+d_{out}) parameters per matrix, 39.98 M in total. On a 24 GB GPU, LoRA (bf16 base) and QLoRA (NF4 base) fit.

In full fine-tuning, optimizer states are 74% of the bill. LoRA trains only the adapter, so its gradients and optimizer states total 0.56 GB beside a frozen bf16 base of 13.5 GB; that is why small changes in rank barely move the LoRA bar. QLoRA stores that base in 4-bit NF4, cutting weights to 3.9 GB.