3 Fine-tuning methods

The chapter introduces fine-tuning as the process of adapting a pretrained language model to a new task by continuing training from existing weights rather than starting from scratch. It explains the core machinery behind this adaptation: how Transformer architectures replaced slower recurrent designs, how tokenization and positional information prepare text for processing, and how self-attention, feed-forward layers, residual connections, and normalization work together to produce contextual representations. It also grounds fine-tuning in gradient descent, showing how loss functions, backpropagation, learning rates, and optimizers like SGD, Adam, AdamW, and Adafactor drive parameter updates during training.

Building on that foundation, the chapter surveys traditional fine-tuning strategies, from supervised fine-tuning with labeled input-output pairs to full-parameter fine-tuning, where every weight is updated, and layer-wise or partial approaches that update only selected parts of the network. These methods are presented as trade-offs between flexibility, cost, and stability: full fine-tuning offers maximum adaptability but risks catastrophic forgetting and heavy compute demands, while selective freezing and layer-wise training reduce memory use and preserve pretrained knowledge. The text emphasizes that the best choice depends on task requirements, dataset size, available hardware, and how much of the model’s general capability must remain intact.

The chapter then turns to parameter-efficient fine-tuning methods that avoid changing the entire model. LoRA adds low-rank adapter matrices to approximate weight updates with far fewer trainable parameters, QLoRA combines this idea with 4-bit quantization and memory-saving optimizers to fit large models on limited hardware, and DoRA refines LoRA by separating weight magnitude from direction for more precise adaptation. NEFTune is introduced as a simpler but effective regularization method that injects noise into embeddings during training to improve instruction-following and reduce overfitting. The chapter closes by broadening the view to unsupervised fine-tuning through domain-adaptive and task-adaptive pretraining, showing how unlabeled text can prepare a model for downstream specialization before any labeled supervision is applied.

Encoder-decoder RNN architecture for language translation. The encoder processes the input sequence sequentially, compressing all information into a single hidden state. The decoder then uses this bottleneck representation to generate the output. A key limitation is that the decoder cannot directly access earlier encoder states. It relies entirely on the compressed final hidden state, which can lead to information loss in longer sequences. Image from Sebastian Raschka’s book Build a Large Language Model (From Scratch) [8].

Transformer model architecture. It consists of an encoder stack (left) and a decoder stack (right). The encoder processes the input sequence using multi-head self-attention (MSA) and multi-layer perceptron (MLP) blocks with residual connections and layer normalization (Add & LN). The decoder processes the target sequence using masked multi-head self-attention (MMSA) and multi-head cross attention (MCA) to integrate encoder features, then projects the output to probabilities through a linear layer and softmax. Source: [9]

The input pipeline before the Transformer block including the positional embedding. Raw text is tokenized into subword units and mapped to integer IDs, which are then looked up in a learned embedding matrix to produce dense token vectors. Positional embeddings encoding each token's position in the sequence are added to the token embeddings, producing the final input matrix passed to the first Transformer block.

The computation is visualized in three stages: (1) Compatibility Scores are generated by multiplying the Queries and Keys; (2) these scores are normalized with a softmax function to produce Attention Weights; (3) the final Context-Aware Output is derived by computing the weighted sum of the Values based on the calculated attention weights.

Gradient descent principle illustrated with a single weight variable. For LLMs, the same principle applies to billions of parameters.

Basic elements of supervised fine-tuning. Starting from a pretrained LLM (the base model), fine-tuning generates a new model using specific knowledge.

Full parameter fine-tuning. The base model is adapted using a domain-specific training dataset, with gradient descent propagating through all layers. The resulting fine-tuned model retains the original architecture but has every weight updated — unlike LoRA or other PEFT methods, nothing is frozen.

In layer-wise fine-tuning, instead of adjusting all the parameters at once, training focuses on one layer at a time. This makes the adaptation more controlled and focused. In the end, all the parameters are tuned.

Only half of the layers are fine-tuned; the others are frozen during training.

LoRA architecture, where the low-rank matrices A and B are fine-tuned while the pre-trained model remains untouched.

The LoRA workflow. Visual breakdown of the fine-tuning process: selecting target layers in a pretrained model, decomposing weights, training low-rank adapters (A′, B′), and merging them back into the final model architecture.

compares LoRA and QLoRA side by side. QLoRA uses a quantized model instead of the original model. It can also offload optimizer states to CPU memory when GPU memory is tight.

DoRA architecture. DoRA improves on standard LoRA by decomposing the pre-trained weight into magnitude and direction components for fine-tuning. LoRA is then applied to efficiently update the direction component. Note that || ||c denotes the vector-wise norm of a matrix across each column vector.

• The Transformer architecture is the backbone of modern language models. It uses self-attention to capture complex contextual dependencies and to build robust pre-trained representations that are later adapted during fine-tuning.
• Gradient descent is the fundamental algorithm behind all fine-tuning. It uses loss functions, backpropagation, and optimizers such as SGD and Adam to iteratively update model parameters.
• Supervised fine-tuning adapts pre-trained models using labeled examples that pair inputs with desired outputs, providing a clear adaptation signal.
• Full-parameter fine-tuning updates every parameter in the model, offering maximum specialization but requiring substantial computational resources and risking catastrophic forgetting.
• Layer-wise fine-tuning (LIFT) progressively tunes one layer at a time, providing controlled adaptation that respects the hierarchical structure of deep networks.
• Half fine-tuning (HFT) and selective layer freezing reduce compute costs by updating only strategic portions of the network (typically the upper layers) while preserving foundational knowledge in the frozen layers.
• Parameter-efficient fine-tuning (PEFT) adds small trainable components instead of updating existing parameters, reducing resource requirements while maintaining performance.
• LoRA decomposes weight updates into low-rank matrices, enabling efficient adaptation with fewer than 1% of the parameters being trainable.
• QLoRA combines LoRA with 4-bit quantization, making large-model fine-tuning feasible on consumer-grade hardware.
• DoRA further refines LoRA by decomposing weights into magnitude and direction, focusing updates where they matter most.
• NEFT adds controlled noise to embeddings during training, improving generalization and reducing overfitting on small datasets.
• Unsupervised fine-tuning uses abundant unlabeled text through domain-adaptive pretraining (DAPT) and task-adaptive pretraining (TAPT), preparing models for specific domains without requiring labeled data.
• All fine-tuning methods involve fundamental trade-offs between depth of adaptation, computational cost, and preservation of the model's original pretrained knowledge.

FAQ