table of content

1 Understanding large language models

1.1 What is an LLM?

1.2 Applications of LLMs

1.3 Stages of building and using LLMs

1.4 Introducing the transformer architecture

1.5 Utilizing large datasets

1.6 A closer look at the GPT architecture

1.7 Building a large language model

2 Working with text data

2.1 Understanding word embeddings

2.2 Tokenizing text

2.3 Converting tokens into token IDs

2.4 Adding special context tokens

2.5 Byte pair encoding

2.6 Data sampling with a sliding window

2.7 Creating token embeddings

2.8 Encoding word positions

3 Coding attention mechanisms

3.1 The problem with modeling long sequences

3.2 Capturing data dependencies with attention mechanisms

3.3 Attending to different parts of the input with self-attention

3.3.1 A simple self-attention mechanism without trainable weights

3.3.2 Computing attention weights for all input tokens

3.4 Implementing self-attention with trainable weights

3.4.1 Computing the attention weights step by step

3.4.2 Implementing a compact self-attention Python class

3.5 Hiding future words with causal attention

3.5.1 Applying a causal attention mask

3.5.2 Masking additional attention weights with dropout

3.5.3 Implementing a compact causal attention class

3.6 Extending single-head attention to multi-head attention

3.6.1 Stacking multiple single-head attention layers

3.6.2 Implementing multi-head attention with weight splits

4 Implementing a GPT model from scratch to generate text

4.1 Coding an LLM architecture

4.2 Normalizing activations with layer normalization

4.3 Implementing a feed forward network with GELU activations

4.4 Adding shortcut connections

4.5 Connecting attention and linear layers in a transformer block

4.6 Coding the GPT model

4.7 Generating text

5 Pretraining on unlabeled data

5.1 Evaluating generative text models

5.1.1 Using GPT to generate text

5.1.2 Calculating the text generation loss

5.1.3 Calculating the training and validation set losses

5.2 Training an LLM

5.3 Decoding strategies to control randomness

5.3.1 Temperature scaling

5.3.2 Top-k sampling

5.3.3 Modifying the text generation function

5.4 Loading and saving model weights in PyTorch

5.5 Loading pretrained weights from OpenAI

6 Fine-tuning for classification

6.1 Different categories of fine-tuning

6.2 Preparing the dataset

6.3 Creating data loaders

6.4 Initializing a model with pretrained weights

6.5 Adding a classification head

6.6 Calculating the classification loss and accuracy

6.7 Fine-tuning the model on supervised data

6.8 Using the LLM as a spam classifier

7 Fine-tuning to follow instructions

7.1 Introduction to instruction fine-tuning

7.2 Preparing a dataset for supervised instruction fine-tuning

7.3 Organizing data into training batches

7.4 Creating data loaders for an instruction dataset

7.5 Loading a pretrained LLM

7.6 Fine-tuning the LLM on instruction data

7.7 Extracting and saving responses

7.8 Evaluating the fine-tuned LLM

7.9 Conclusions

7.9.1 What’s next?

7.9.2 Staying up to date in a fast-moving field

7.9.3 Final words

Appendixes

Appendix A: Introduction to PyTorch

A.1 What is PyTorch?

A.1.1 The three core components of PyTorch

A.1.2 Defining deep learning

A.1.3 Installing PyTorch

A.2 Understanding tensors

A.2.1 Scalars, vectors, matrices, and tensors

A.2.2 Tensor data types

A.2.3 Common PyTorch tensor operations

A.3 Seeing models as computation graphs

A.4 Automatic differentiation made easy

A.5 Implementing multilayer neural networks

A.6 Setting up efficient data loaders

A.7 A typical training loop

A.8 Saving and loading models

A.9 Optimizing training performance with GPUs

A.9.1 PyTorch computations on GPU devices

A.9.2 Single-GPU training

A.9.3 Training with multiple GPUs

Appendix B: References and further reading

Appendix C: Exercise solutions

Appendix D: Adding bells and whistles to the training loop

D.1 Learning rate warmup

D.2 Cosine decay

D.3 Gradient clipping

D.4 The modified training function

Appendix E: Parameter-efficient fine-tuning with LoRA

E.1 Introduction to LoRA

E.2 Preparing the dataset

E.3 Initializing the model

E.4 Parameter-efficient fine-tuning with LoRA

Overview

Appendix D. Adding Bells and Whistles to the Training Loop

This appendix refines the training loop used for pretraining and finetuning by adding practical stability and convergence improvements. To keep the material self-contained, it reinitializes the GPT model and reconstructs the data loaders from earlier chapters, using a short story dataset for demonstration. The overarching goal is to introduce and then integrate three techniques—learning rate warmup, cosine decay, and gradient clipping—into a cohesive training function that can reliably pretrain a small LLM.

The first enhancement is learning rate warmup, which starts training with small updates and linearly raises the learning rate from an initial value to a user-defined peak over a fixed number of steps (for example, from 0.0001 to 0.01 over 20 steps). After warmup, a cosine decay schedule gradually reduces the learning rate from its peak toward a small floor (for example, 0.1 times the initial learning rate) following a half-cosine curve across the remaining training steps. Together, these schedules help avoid destabilizing updates early on and reduce the risk of overshooting minima later by slowing the learning rate as training progresses.

The appendix also adds gradient clipping to curb exploding gradients by constraining the L2 norm of all parameter gradients to a specified maximum (for example, max_norm = 1.0 via clip_grad_norm_). The text explains the L2 norm notionally and demonstrates, with a simple measurement of per-parameter gradients before and after clipping, that the largest gradients are reduced to a safer range. In the final step, all three techniques—warmup, cosine decay, and gradient clipping—are combined within the training loop to stabilize and improve LLM pretraining.

Figure D.1 The learning rate warmup increases the learning rate for the first 20 training steps. After 20 steps, the learning rate reaches the peak of 0.01 and remains constant for the rest of the training.

Figure D.2 The first 20 steps of linear learning rate warmup are followed by a cosine decay, which reduces the learning rate in a half-cosine cycle until it reaches its minimum point at the end of training.

D.4 The modified training function

In this final section of this appendix, we improve the train_model_simple training function we used in chapter 5 by adding the three concepts we introduced: linear warmup, cosine decay, and gradient clipping. Together, these methods help stabilize LLM training.

The code is as follows, with the changes compared to the train_model_simple annotated:

After defining the train_model function, we can use it in a similar fashion to train the model compared to the train_model_simple method in chapter 5:

The training will take about 5 minutes to complete on a MacBook Air or similar laptop and print the following outputs:

Like chapter 5, the model begins to overfit after a few epochs since it is a very small dataset, and we iterate over it multiple times. However, we can see that the function is working since it minimizes the training set loss.

Readers are encouraged to train the model on a larger text dataset and compare the results obtained with this more sophisticated training function to the results that can be obtained with the train_model_simple function used in chapter 5.

FAQ

What is learning rate warmup and why is it useful for training LLMs?

Learning rate warmup gradually increases the learning rate from a small initial value (initial_lr) to a user-defined peak_lr over the first warmup_steps. This reduces the chance of destabilizing, large parameter updates early in training, which is especially helpful for large, deep models like LLMs.

How do I implement a simple linear warmup schedule in PyTorch?

Compute a per-step increment lr_increment = (peak_lr - initial_lr) / warmup_steps. Track a global_step that starts at -1 and increments once per optimizer update. If global_step < warmup_steps, set lr = initial_lr + global_step * lr_increment; otherwise set lr = peak_lr. Apply the value to each optimizer parameter group via for pg in optimizer.param_groups: pg["lr"] = lr.

How should I choose the number of warmup steps and the peak learning rate?

Warmup steps are often a small fraction of total steps (for example, 1–5%), but this depends on model size, batch size, and optimizer. The peak_lr should be the highest stable learning rate for your setup; err on the side of smaller values for larger models or noisier data and increase cautiously while monitoring loss stability.

What is cosine decay and how does it change the learning rate after warmup?

Cosine decay smoothly reduces the learning rate from peak_lr toward a minimum value (min_lr) following a half-cosine curve after warmup. It slows updates later in training to avoid overshooting minima and to encourage stable convergence.

What is the formula used for cosine decay in this appendix?

After warmup, compute progress = (global_step - warmup_steps) / (total_training_steps - warmup_steps), clamped to [0, 1]. Then set lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + cos(pi * progress)). Before warmup ends, use the linear warmup rule.

Why set a minimum learning rate (min_lr), and why 0.1 × initial_lr in the example?

A minimum learning rate prevents the scheduler from driving the LR to exactly zero, which can stall learning or cause numerical issues. Using min_lr = 0.1 × initial_lr is a simple, conservative floor; it’s a heuristic and can be tuned based on validation performance.

How do I compute total_training_steps and track the schedule?

Set total_training_steps = len(train_loader) * n_epochs. Maintain a global_step counter that increments once per optimizer update. Record the current optimizer.param_groups[0]["lr"] each step (for example, append to a list) and visualize it to verify the schedule.

What is gradient clipping and when should I apply it in the training loop?

Gradient clipping limits the magnitude (norm) of gradients to a maximum value (max_norm) to prevent exploding updates. In PyTorch, call torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=...) after loss.backward() and before optimizer.step().

What’s the difference between clip_grad_norm_ and clip_grad_value_?

clip_grad_norm_ rescales gradients so their global L2 norm does not exceed max_norm, keeping direction while reducing magnitude. clip_grad_value_ clamps individual gradient values to a specified range. For LLMs, norm-based clipping (clip_grad_norm_) is more commonly used.

How do I combine warmup, cosine decay, and gradient clipping in a complete training step?

Typical order per step: set model.train(); optimizer.zero_grad(); compute lr using warmup/cosine schedule and assign to optimizer.param_groups; forward pass and compute loss; loss.backward(); torch.nn.utils.clip_grad_norm_(...); optimizer.step(); update global_step; optionally log lr and loss.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

eBook

pdf, ePub, online

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more