Appendix B. References and Further Reading
This appendix curates a broad map of the LLM landscape, from the foundational ideas behind transformers to practical resources for building, training, and adapting models. It highlights the rise of decoder-style GPT models, the continued relevance of encoder-style transformers for classification, and the extension of transformer ideas beyond text to vision. It also showcases how domain-specific pretraining and task-focused finetuning can surpass general-purpose models, while noting emerging non-transformer architectures that challenge the status quo. Overall, it serves as a guide to both the conceptual pillars and the hands-on tools needed to construct and refine large language models from scratch.
On the architectural side, the references trace the lineage from scaled dot-product attention to modern GPT systems, alongside BERT and the vision transformer, and include alternatives such as RWKV, Hyena, and state-space models like Mamba. Efficiency and stability are recurring themes, with entries on FlashAttention, PyTorch attention primitives, dropout, and normalization strategies (LayerNorm, Pre-LN vs. Post-LN, RMSNorm), as well as GELU activations. The appendix also points to tokenizer design and tooling (BPE, SentencePiece, WordPiece), scaling insights from GPT-2 to GPT-3, and instruction tuning with human feedback. Collectively, these works ground the reader in the core components and trade-offs that underpin modern LLMs.
On the practical front, the appendix aggregates resources for datasets, training curricula, and evaluation. It spans large open corpora and suites for analyzing scaling (Dolma, The Pile, RefinedWeb, RedPajama, FineWeb, Pythia, OLMo), data preparation pipelines (for example, converting public-domain books for pretraining), and training strategies such as warmup, cosine annealing, continued pretraining, and memory-efficient optimization via low-rank gradient projection (GaLore). Decoding and inference references cover methods like top-k sampling and beam search. Finetuning materials include tutorials and experiments on classification with decoder and encoder models, handling class imbalance, layer-freezing choices, and modifications that improve classification (e.g., removing causal masks and deriving powerful text embeddings). Case studies in finance and medicine illustrate the gains from domain adaptation and task-specific tuning, rounding out a practical toolkit for building and customizing LLMs.
Chapter 7
The Alpaca dataset for instruction finetuning contains 52 thousand instruction-response pairs and is one of the first and most popular publicly available datasets for instruction finetuning:
- Stanford Alpaca: An Instruction-following Llama Model, https://github.com/tatsu-lab/stanford_alpaca
Listed below are additional publicly accessible datasets suitable for instruction finetuning:
- LIMA, https://huggingface.co/datasets/GAIR/lima; one thousand high quality instruction-response pairs; for more information, see the LIMA: Less Is More for Alignment (2023) paper, https://arxiv.org/abs/2305.11206
- UltraChat, https://huggingface.co/datasets/openchat/ultrachat-sharegpt; a large-scale dataset consisting of 805 thousand instruction-response pairs; for more information, see the Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (2023) paper, https://arxiv.org/abs/2305.14233
- Alpaca GPT4, https://github.com/Instruction-Tuning-with-GPT-4/GPT-4-LLM/blob/main/data/alpaca_gpt4_data.json, an Alpaca-like dataset with 52 thousand instruction-response pairs generated with GPT-4 instead of GPT-3.5
Phi-3 is a 3.8B parameter model with an instruction-finetuned variant that is reported to be comparable to much larger proprietary models, such as GPT-3.5:
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024) by Abdin et al., https://arxiv.org/abs/2404.14219
Researchers propose a synthetic instruction data generation method that generates 300,000 high-quality instruction-response pairs from an instruction-finetuned Llama-3 model. A pretrained Llama 3 base model finetuned on these instruction examples performs comparably to the original instruction-finetuned Llama-3 model:
- Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024) by Xu et al., https://arxiv.org/abs/2406.08464
Research has shown that not masking the instructions and inputs in instruction finetuning effectively improves performance on various NLP tasks and open-ended generation benchmarks, particularly when trained on datasets with lengthy instructions and brief outputs or when using a small number of training examples:
- Instruction Tuning With Loss Over Instructions (2024) by Shi, https://arxiv.org/abs/2405.14394
Prometheus and PHUDGE are openly available LLMs that match GPT-4 in evaluating long-form responses with customizable criteria. We haven't used these in chapter 7 because they are not yet supported by Ollama, and thus cannot be executed efficiently on a laptop.
- Prometheus: Inducing Finegrained Evaluation Capability in Language Models (2023) by Kim et al., https://arxiv.org/abs/2310.08491
- PHUDGE: Phi-3 as Scalable Judge (2024) by Deshwal and Chawla, https://arxiv.org/abs/2405.08029
- Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024), https://arxiv.org/abs/2405.01535
The results in the following report support the view that large language models primarily acquire factual knowledge during pretraining and that finetuning mainly enhances their efficiency in using this knowledge. Furthermore, this study explores how finetuning large language models with new factual information affects their ability to use preexisting knowledge, revealing that models learn new facts more slowly and their introduction during finetuning increases the model's tendency to generate incorrect information:
- Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (2024) by Gekhman, https://arxiv.org/abs/2405.05904
Preference finetuning is an optional step after instruction finetuning to align the LLM more closely with human preferences. The following articles by the author provide more information about this process:
- LLM Training: RLHF and Its Alternatives, https://magazine.sebastianraschka.com/p/llm-training-rlhf-and-its-alternatives
- Tips for LLM Pretraining and Evaluating Reward Models, https://sebastianraschka.com/blog/2024/research-papers-in-march-2024.html
Build a Large Language Model (From Scratch) ebook for free
