This appendix consolidates concise, code-centered solutions to the book’s exercises, spanning tokenization, data loading, attention mechanisms, model scaling, sampling strategies, evaluation, finetuning, prompting, and parameter-efficient training. It emphasizes practical snippets and measured outcomes so readers can reproduce behaviors and understand trade-offs. The accompanying implementations are referenced in a supplementary repository, and the text highlights when particular settings impact speed, memory, or accuracy.
Early exercises focus on foundations: obtaining specific token IDs via per-string encoding, verifying reconstruction with decoding, and illustrating how max_length and stride shape batches in a sliding-window data loader. Attention exercises map weight assignments between implementations, show how to match output dimensionality in multi-head attention by adjusting projection size, and provide initialization settings for GPT-2 small. Transformer block accounting demonstrates that the feed-forward module contains roughly twice as many parameters as attention. The appendix also shows how to scale configurations to GPT-2 XL, reporting parameter counts on the order of 1.6B and corresponding memory footprint.
Later sections investigate generation and finetuning behaviors. They quantify how temperature and top-k influence randomness (with options for fully deterministic decoding), continue training from checkpoints, and compare train/validation losses with plausible explanations for their proximity. Several finetuning experiments show that excessive padding hurts accuracy, updating all layers can yield modest gains, and relying on the first token degrades performance. Prompting with a Phi-3-style template shortens inputs and speeds training by about 17% with comparable scores, while instruction masking slightly reduces performance. Switching to the larger Alpaca dataset requires GPU-friendly settings, and a LoRA-based approach cuts trainable parameters and accelerates training by roughly 28% while maintaining similar evaluation results.
FAQ
Where can I find the complete code for the exercise solutions?The full, runnable code for all exercise answers is in the supplementary GitHub repository: https://github.com/rasbt/LLMs-from-scratchHow do I obtain individual token IDs for substrings and reconstruct the original text?Encode each substring separately with tokenizer.encode and then decode the collected IDs with tokenizer.decode. Example: tokenizer.encode("Ak"), tokenizer.encode("w"), ...; later, tokenizer.decode([id1, id2, ...]) to reassemble the string.How do max_length and stride affect the data loader batches?They control the window size and overlap of token sequences. For example, create_dataloader(raw_text, batch_size=4, max_length=2, stride=2) yields non-overlapping pairs of tokens per row, while max_length=8 with stride=2 yields overlapping 8-token windows that advance by 2 tokens, producing more context and overlap across rows.How do I correctly copy weights between two self-attention implementations with different parameter shapes?Transpose the source Linear weights when assigning to Parameter tensors. For example: sa_v1.W_query = torch.nn.Parameter(sa_v2.W_query.weight.T), and likewise for key and value.How can a 2-head multi-head attention return a 2-dimensional output like single-head attention?Set the per-head projection dimension to 1 so the concatenated output is 2. Example: d_out = 1; mha = MultiHeadAttentionWrapper(d_in, d_out, block_size, 0.0, num_heads=2).Why does the feed-forward module have about twice as many parameters as the attention module?In the 124M configuration’s TransformerBlock, summing parameters shows roughly 4,722,432 in the feed-forward vs. 2,360,064 in attention—about 2×. The feed-forward layers expand and project the embedding dimension, dominating parameter count compared to Q/K/V projections plus output projection in attention.How do I instantiate a GPT-2 XL–sized configuration and how large is it?Start from the 124M config, then set emb_dim=1600, n_layers=48, n_heads=25. The resulting model has about 1.64B parameters (about 1.56B trainable with weight tying) and is roughly 6.25 GB in size (float32).How should I set temperature and top-k, and how can I make generation deterministic?- Lower temperature (≤1) and small top-k (e.g., <10) reduce randomness and yield more deterministic, coherent outputs; higher temperature (>1) and larger top-k (≈20–40) increase creativity. - To force determinism: either set top_k=1, or use top_k=None with no temperature scaling (temperature=1 and sample from argmax).How do I resume training from a saved checkpoint?Load the saved model and optimizer state dicts, then continue training. Example: checkpoint = torch.load("model_and_optimizer.pth"); model.load_state_dict(checkpoint["model_state_dict"]); optimizer.load_state_dict(checkpoint["optimizer_state_dict"]); then call your train function for additional epochs.How can I add LoRA for instruction finetuning, and what benefits should I expect?Freeze base model params, replace Linear layers with LoRA adapters (e.g., rank=16, alpha=16) using helper utilities from appendix E, and train only the LoRA parameters. On an Nvidia L4 GPU, this ran about 28% faster with comparable evaluation scores to the full finetune in the chapter.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Build a Large Language Model (From Scratch) ebook for free
