What Is LoRA Rank?

LoRA rank (r) is the inner dimension of the two small matrices LoRA trains instead of a full weight update. It sets how many independent directions an adapter can push the frozen weights in, making it the capacity dial: too low and the model cannot fit your task, too high and you lose LoRA’s efficiency.

What does the rank actually control?

Rank controls the expressiveness of the weight update, not the size of the model. LoRA freezes a pretrained weight matrix W and learns two matrices A and B such that the effective weight becomes W + BA. If W is d × k, then B is d × r and A is r × k. Every update the adapter can represent therefore has rank at most r.

The LoRA authors argue that adaptation updates are intrinsically low-rank — that most of the change a downstream task needs lives in a small subspace. That is why a rank far below the full matrix dimension still tracks full fine-tuning quality. Against GPT-3 175B tuned with Adam, they report reducing trainable parameters by 10,000 times and GPU memory by 3 times, with quality on par with or better than full fine-tuning.

How many parameters does a given rank add?

The arithmetic is exact: a LoRA pair on a d × k matrix adds r × (d + k) trainable parameters. For a square 2048 × 2048 projection — a realistic hidden size for a 1B-class student — that works out as follows.

Rank r	Trainable parameters added	Share of the full 2048 × 2048 matrix
4	16,384	0.4%
8	32,768	0.8%
16	65,536	1.6%
32	131,072	3.1%
64	262,144	6.3%

Two things fall out of this table. Parameter count grows linearly in r, so doubling the rank doubles the adapter but never approaches the cost of the frozen matrix. And even at r = 64 you are training a small fraction of one layer’s weights, which is why adapter files ship as megabytes rather than gigabytes.

How do you choose a rank?

Start from the platform default and only move if evaluation tells you to. On distil labs the configuration file sets lora_r to 64 with use_lora enabled by default, which is a deliberately generous rank for the small students in the supported models catalog.

Raise the rank when the task requires genuinely new behaviour — an unfamiliar output schema, a domain vocabulary the base model has never seen — and the held-out scores plateau below your bar. Lower it when the model reproduces training examples but fails on paraphrases, which is a capacity-driven memorisation symptom covered in what is overfitting in fine-tuning.

Be sceptical of rank as a first resort. The PEFT survey by Lialin et al. compared 15 parameter-efficient methods on models up to 11B and found that methods claimed to beat a strong LoRA baseline struggled once hyperparameter search was limited and training ran for only a few epochs. In a constrained setting, data quality moves the number more than rank does.

What is LoRA alpha and how does it relate to rank?

Alpha is a scaling factor applied to the low-rank update, and it is defined relative to rank rather than independently. distil labs exposes it as lora_alpha_multiplier, documented as lora_r * lora_alpha_multiplier, with a default multiplier of 1 — so alpha tracks whatever rank you set.

That coupling matters: if you change lora_r and leave the multiplier alone, the effective scaling of the update changes with it. Keeping the multiplier at its default is the reason you can adjust rank in isolation without re-tuning the learning rate.

Related terms

Term	How it relates to rank
Adapter	The trained BA pair itself; rank is its width. See full fine-tuning vs adapters.
QLoRA	Quantises the frozen base to 4-bit and keeps the same rank-r adapters. See LoRA vs QLoRA.
Full fine-tuning	The r = full dimension limit, where every weight moves. See LoRA vs full fine-tuning.
Target modules	Which matrices get an adapter at all; multiplies the per-matrix cost above.

The Hugging Face PEFT library is the reference implementation for all of these, and its r argument is the same quantity described here.