16 Text generation

This chapter surveys how generative modeling moved from niche curiosity to everyday tool and argues for AI as augmented intelligence in creative and practical workflows. It traces sequence generation from early LSTMs and Graves’ handwriting systems to the Transformer era, highlighting how OpenAI’s GPT series showed that large-scale next-token pretraining plus compute can unlock strong zero- and few-shot behaviors—alongside persistent issues like hallucinations, prompt sensitivity, and limited out‑of‑distribution robustness. With that context, the chapter positions large language models as foundation models trained to learn statistical structure and sample from latent spaces, reshaping how text (and beyond) can be produced and steered.

Readers then build a mini GPT from scratch: preparing web text (C4 subset) with subword tokenization, assembling a decoder-only Transformer with causal attention, positional embeddings with tied input/output projections, and stability measures such as warmup schedules and mixed precision. The chapter demonstrates generative decoding pitfalls and remedies (compiled inference, KV caching) and explores sampling strategies—greedy, temperature, Top‑K, and beam—to balance fluency and diversity. Because small models remain limited, it transitions to using a billion‑parameter pretrained Gemma model, showing prompting behaviors, instruction fine‑tuning to align outputs, and Low‑Rank Adaptation (LoRA) to drastically reduce trainable memory for practical fine‑tuning on commodity GPUs.

Finally, it broadens to methods that make LLMs more useful and reliable: Reinforcement Learning with Human Feedback (RLHF) to align models with preferences; instruction‑tuned checkpoints for chat; multimodal extensions that splice learned “soft” image tokens into text sequences; and Retrieval‑Augmented Generation (RAG) to ground answers in up‑to‑date or private sources. It explores emerging “reasoning” workflows (chain‑of‑thought prompting and self‑training on verifiable tasks like math and code), and discusses scaling trends, costs, energy and data constraints, and the shift toward foundation models adapted to downstream tasks. The chapter closes with a realistic outlook: today’s recipe—large‑scale self‑supervision, targeted fine‑tuning, and tool‑augmented prompting—yields powerful systems, yet open challenges in reliability, efficiency, and controllability remain central to the field’s next steps.

An image generated with the generative image software Midjourney. The prompt was A hand-drawn, sci-fi landscape of residents living in a building shaped like a red letter K.

Warmup makes our updates to model parameters smaller at the beginning of training, and can help with stability.

Greedy, Top-K and Random sampling strategies shown on the same probability distribution.

The low-rank kernel decomposition contains far fewer parameters than the kernel itself.

LoRA greatly reduces the memory we need for gradients and optimizer states.

Handling image input by splicing text tokens and “soft” image tokens together.

A test image for the Gemma model. HTML: class=“small-image”

LLM parameter counts (left) and pretraining dataset sizes (right) over time. Note that many recent proprietary LLMs (e.g. GPT-4, Gemini), are not included because model details have not been disclosed.

• Large Language Models, or LLMs are the combination of a few key ingredients:
  • The Transformer architecture.
  • A language modeling task (predicting the next token based on past tokens).
  • A large amount of unlabeled text data.
• An LLM learns a probability distribution for predicting individual tokens. This can be combined with a sampling strategy to generate a long string of text. There are many popular ways to sample text:
  • Greedy search takes the most likely predicted token at each generation step.
  • Random sampling directly samples the predicted categorical distribution over all tokens.
  • Top-K sampling restricts the categorical distribution to the top set of K candidates.
• LLMs use billions of parameters and are trained on trillions of words of text.
• LLM output is unreliable, and all LLMs will occasionally hallucinate factually incorrect information.
• LLMs can be fine-tuned to follow instructions in a chat dialog. This type of fine-tuning is called instruction fine-tuning.
  • The simplest form of instruction fine-tuning involves directly training the model on instruction and response pairs.
  • More advanced forms of instruction fine-tuning involve reinforcement learning.
• The most common resource bottleneck when working with LLMs is accelerator memory.
  • LoRA is a technique to reduce memory usage by freezing most Transformer parameters and only updating a low-rank decomposition of attention projection weights.
• LLMs can input or output data from different modalities if you can figure out how to frame these inputs or outputs as sequences in a sequence prediction problem.
  • A foundation model is a general term for models, of any modality, trained using self-supervision for a wide range of downstream tasks.

FAQ