Full fine-tuning is the luxury the rest of us cannot afford. A 12B-parameter model in fp16 needs around 47GB of GPU memory before the optimizer even gets its share, which is why so many promising fine-tuning projects died the moment someone loaded the weights. huggingface/peft, the Parameter-Efficient Fine-Tuning library, is the industry’s standard answer: freeze the backbone, learn a tiny set of extra weights instead, and ship checkpoints measured in megabytes. The Apache-2.0 repository has grown into a zoo of 46 tuner methods, from LoRA and DoRA through prompt tuning, IA3, VeRA, and X-LoRA, all sharing one injection framework and one checkpoint format. It is the library underneath QLoRA training jobs, TRL reinforcement-learning runs, and countless adapter swaps on the Hub. As always in this series, what follows is an educational tour of published source code.

The codebase is organized with unusual discipline for something this dynamic. The public entry point is a single function, get_peft_model, in an 81-line dispatch module that maps a config object to a tuner class through registries that methods register themselves into. The wrapper layer, peft_model.py, runs 3242 lines and adds six task-specific subclasses on top of the base PeftModel. Underneath, every tuner follows the same contract defined in tuners_utils.py: walk the model, find target modules, replace them with wrapped layers. The LoRA implementation is the reference citizen, with a 2429-line layer file that carries the full math of low-rank adaptation, including rank-stabilized scaling, per-layer rank patterns, DoRA magnitude vectors, and merge and unmerge surgery on quantized weights. Reading the overview from left to right shows a clean funnel: two entry functions, a registry, one big wrapper family, and one tuner contract.

Overview of PEFT: two entry functions share a registry of tuner classes, PeftModel and its six task subclasses wrap the result, and the LoRA tuner demonstrates the layer contract that all 46 methods follow.

Reading the overview from left to right:

• get_peft_model is the front door: it resolves a config such as LoraConfig into a tuner, injects the adapters, and hands back a PeftModel.
• inject_adapter_in_model is the lower-level sibling that mutates your model in place and returns the same object, for people who already have their own wrapper.
• register_peft_method is how the 46 methods plug in: each tuner registers its config class, tuner class, and checkpoint prefix into shared tables.
• PeftModel is the base wrapper at line 140, carrying adapter add, load, set, and delete operations plus save_pretrained and generate.
• six task subclasses specialize forward and generate for causal LM, seq2seq, classification, token classification, question answering, and feature extraction.
• LoraModel is the reference tuner: it walks named_modules and swaps every target for a LoRA-wrapped copy, honoring per-key rank and alpha patterns.
• LoraLayer holds the math: per-adapter rank, the A and B matrices, the alpha-over-r scaling, and the merge code that folds adapters into base weights.
• save_and_load.py is why checkpoints stay tiny: it filters the state dict down to adapter weights only, keyed by adapter name.

Why You Need This

The README opens with a table that explains the entire value proposition in three rows. On an A100 80GB GPU, full fine-tuning of bigscience/T0_3B consumes 47.14GB of GPU memory; the same model with LoRA needs 14.4GB; LoRA plus DeepSpeed CPU offloading brings it to 9.8GB. Move up to the 12B mt0-xxl and full fine-tuning runs out of memory entirely, while LoRA fits at 56GB and offloaded LoRA at 22GB. The bloomz-7b1 numbers follow the same shape. Quality holds up too: on the twitter_complaints task, the LoRA-trained T0 scores 0.863 against a Flan-T5 baseline of 0.892 and a crowdsourced human baseline of 0.897.

Storage is the second argument, and it is the one that changed how people share models. A full T0_3B checkpoint is 11GB; the LoRA adapter for the same model is 19MB, roughly a 500-fold reduction. That ratio is what makes the adapter economy on the Hugging Face Hub possible: one frozen backbone on disk, hundreds of task adapters beside it, each a few megabytes. The library leans into this with load_adapter and set_adapter on PeftModel, so a serving process can hot-swap personalities without reloading weights, and with create_or_update_model_card, which writes a metadata card into every checkpoint folder it saves.

The third argument is the quantization story that became QLoRA. The LoRA tuner ships dedicated wrapper files for bitsandbytes 4-bit and 8-bit layers, plus GPTQ, AQLM, AWQ, HQQ, and EETQ formats, each teaching the adapter how to dequantize, add the low-rank update, and merge back without ever materializing the full fp16 weight. The README points to a recipe that fine-tunes meta-llama/Llama-2-7b-hf on a 16GB consumer GPU using exactly this path. If you have ever wondered what actually happens between “load_in_4bit” and a trainable layer, this is the code that answers it.

Detail view: the entry and registry layer, the PeftModel family with its adapter lifecycle, the LoRA internals from config to scaling to merge, and the quantized variants plus checkpoint format.

Start at the entry points. get_peft_model, defined at line 105 of mapping_func.py, and inject_adapter_in_model, at line 47 of mapping.py, both depend on tables that start empty: PEFT_TYPE_TO_CONFIG_MAPPING, PEFT_TYPE_TO_TUNER_MAPPING, and friends are dicts in mapping.py that register_peft_method fills at import time. That registration pattern is why the library can carry 46 methods without a single if-chain in its dispatch code; adding a tuner means adding a folder and a registration call, never touching the front door. The injection function does enforce a few hard rules, rejecting prompt-learning methods and tuners with shared state, since those need the full PeftModel wrapper rather than bare mutation.

PeftModel, at line 140 of peft_model.py, is where the wrapper personality lives. Its constructor branches on whether the config is prompt-learning: prompt methods keep the raw model and hang a prompt encoder off it, while everything else instantiates the tuner class from the registry, which returns the model with wrapped layers inside. After wrapping it casts adapter dtypes, prepares gradient checkpointing, and forces pretraining_tp to 1 to avoid numerical surprises from tensor-simulation settings. The adapter lifecycle is explicit and worth knowing for serving: add_adapter at line 1109, load_adapter at 1515, set_adapter at 1705, and delete_adapter at 1190. Meanwhile get_nb_trainable_parameters at line 953 powers print_trainable_parameters, the call that prints the famous line from the README quickstart: 3,686,400 trainable parameters out of 3,089,625,088 total, or 0.1193 percent, for a Qwen2.5-3B model with rank 16.

The LoRA internals repay a slow read. LoraLayer, at line 108 of layer.py, keeps per-adapter dicts of rank, the A and B matrices, dropout, and scaling, so multiple adapters coexist in one layer. In update_layer, lora_A is created as a plain nn.Linear from in_features down to rank r, and lora_B as an nn.Linear from r back up to out_features; A gets kaiming-uniform initialization while B starts at zero, which is the trick that keeps the initial adapter output silent. The scaling factor is lora_alpha divided by r at line 281, or alpha over the square root of r when use_rslora is set, following the rank-stabilized recipe. Forward passes then compute base_layer(x) plus the scaled B-times-A projection, and the merge code computes the delta weight as exactly that product so adapters can be folded into the base matrix and later unfolded without loss.

The traversal side is LoraModel, at line 165 of model.py. Its _create_and_replace at line 294 resolves per-key overrides first: rank_pattern and alpha_pattern are dicts of regex-to-value, looked up with get_pattern_key against the current module name, so you can ask for rank 64 on attention projections and rank 8 everywhere else in one config. It then passes r, alpha, and quantization flags into _create_new_module at line 499, which picks the right wrapper class for the target, including the quantized variants. If the target is already a LoRA layer, it simply adds another adapter to it, which is how multi-adapter layers work. The config file itself is a study in accretion: LoraConfig at line 514 is surrounded by companion configs for runtime options, VeRA, LoftQ, ARROW, EVA, CorDA, and more, each representing a research refinement the community adopted.

Persistence is the quiet achievement. get_peft_model_state_dict at line 148 of save_and_load.py extracts only adapter weights, remaps their names into the checkpoint format, and set_peft_model_state_dict at line 549 reverses the process on load, with load_peft_weights at 702 handling the remote fetch from the Hub. The artifact is two files, adapter_model.safetensors and adapter_config.json, the name defined in constants.py at line 400. Because every method uses the same keys, a checkpoint carries its own config and can be loaded years later without the original training script. For debugging, TunerLayerStatus and TunerModelStatus at lines 3358 and 3488 print per-layer tables of what is wrapped where, and mixed_model.py, at 397 lines, composes several methods into one model for the rare cases where LoRA plus prompt tuning is genuinely the answer.

From Install to a Trained Adapter

The quickstart from the README trains a rank-16 LoRA adapter on a 3B model in a few lines:

pip install peft

python - <<'PY'
import torch
from transformers import AutoModelForCausalLM
from peft import LoraConfig, TaskType, get_peft_model

device = torch.accelerator.current_accelerator().type if hasattr(torch, "accelerator") else "cuda"
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", device_map=device)
peft_config = LoraConfig(r=16, lora_alpha=32, task_type=TaskType.CAUSAL_LM)
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
# trainable params: 3,686,400 || all params: 3,089,625,088 || trainable%: 0.1193

# train with your favorite Trainer, then:
model.save_pretrained("qwen2.5-3b-lora")
PY

Loading for inference mirrors it: rebuild the base model, then PeftModel.from_pretrained(model, “qwen2.5-3b-lora”) reattaches the adapter and you can generate immediately.

Honest limits: PEFT is a wrapper library, not a trainer, so you still bring Transformers, Accelerate, or TRL for the training loop, and the README’s own examples lean on those siblings for scheduling and distributed setup. Prompt-learning methods and tuners with shared state cannot use the in-place injection API, a restriction the code states rather than hides. The 46-method zoo is a strength and a hazard: method quality varies by task and model, and the README is candid that numbers like the 0.863 accuracy come from an unoptimized configuration you can probably beat with tuning. Merging adapters into base weights is lossless only when you control the dtype math, and on quantized backbones the merge path is necessarily heavier. None of that diminishes the core result this repository proves line by line: for a very large class of adaptation problems, the 99.88 percent of the model you were about to train did not need training at all.

Enjoyed this post? Never miss out on future posts by following us