11 Building A Generative Pre-Trained Transformer (GPT) from Scratch

This chapter introduces how to build a generative pre-trained Transformer in the GPT family entirely from scratch in PyTorch and use it to generate text. It motivates the design through GPT-2, a landmark 2019 model that popularized large-scale next-token prediction and sparked discussions about safety and staged releases. The narrative balances capabilities with caveats, contrasting GPT-2XL’s 1.5B parameters with far larger successors, and frames expectations around coherence versus true understanding. The chapter’s goals are to demystify the decoder-only architecture, explain causal self-attention, and culminate in extracting and loading public GPT-2XL weights so the hand-built model can produce coherent continuations from prompts.

Architecturally, the model is a decoder-only Transformer: token embeddings and learned positional embeddings are summed and passed through a deep stack of identical blocks. Each block combines causal multi-head self-attention with a position-wise feed-forward network using GELU, wrapped in layer normalization and residual connections for stable training. Causality is enforced by masking future positions so each token attends only to itself and its predecessors. The chapter also covers subword tokenization with BPE, the large GPT-2 vocabulary, and fixed sequence length, all geared toward next-token prediction on broad web text. Together, these choices let the model capture local and medium-range dependencies while remaining amenable to parallel computation.

Practically, you implement the components step by step: a configuration object, a CausalSelfAttention module that registers the causal mask as a buffer, a Transformer block, and the full GPT-2XL. You then fetch pre-trained GPT-2XL weights via a standard library, adapt checkpoint tensors where the original uses Conv1d-style parameter shapes by transposing them, and load them into your model. With a simple generate pipeline—temperature and top-k sampling, iterative decoding, and an end-of-text stop token—you can produce coherent continuations and explore creativity-versus-consistency trade-offs. The chapter closes by emphasizing limits: GPT-2 can be fluent yet fabricate facts, degrades over longer contexts, and lacks true semantic comprehension, and training such a large model from scratch is infeasible without substantial compute, making weight loading essential for hands-on experimentation.

The architecture of the GPT-2 model. GPT-2 is a decoder-only Transformer, consisting of N identical decoder layers. Each decoder block contains two sublayers. The first sublayer is a causal self-attention layer. The second is a feed-forward network. Each sublayer uses layer normalization and a residual connection. The input is first passed through word embedding and positional encoding, and the sum is then passed through the decoder. The output from the decoder goes through layer normalization and a linear layer.

GPT-2 first represents each token in a sequence with a 50276-value one-hot vector. The token representation of the sequence goes through a word embedding layer to compress it into an embedding with a dimension of 1600. GPT-2 also represents each position in a sequence with a 1024-value one-hot vector. The positional representation of the sequence goes through a positional encoding layer to compress it into an embedding also with a dimension of 1600. The word embedding and positional encoding are added together to form the input embedding.

How to calculate masked attention weights in causal self-attention. A mask is applied to the scaled attention scores so that values corresponding to future tokens (those above the main diagonal in the matrix) become -∞. We then apply the softmax function on the masked scaled attention scores and obtain the masked attention weights. The masking ensures that predictions for a given token can only be influenced by the tokens that precede it in the sequence, not by future tokens. This is essential for generating coherent and contextually relevant text outputs.

Comparing the GELU activation function with ReLU. The solid line is the ReLU activation function, while the dashed line is the GELU activation function. ReLU is not differentiable everywhere since there is a kink in it. GELU, in contrast, is differentiable everywhere. This smoothness in GELU helps in optimizing the neural network more effectively, as it provides a more continuous gradient for backpropagation during the training process.

• Generative Pre-trained Transformer 2 (GPT-2) is an advanced large language model (LLM) developed by OpenAI and announced in February 2019. It represents a significant milestone in the field of natural language processing (NLP) and has paved the way for the development of even more sophisticated models, including its successors, ChatGPT and GPT-4.
• GPT-2 is a decoder-only Transformer, meaning there is no encoder stack in the model. Like other Transformer models, GPT-2 uses self-attention mechanisms to process input data in parallel, significantly improving the efficiency and effectiveness of training LLMs.
• GPT-2 adopts a different approach to positional encoding than the one used in the seminal 2017 paper, Attention Is All You Need. Instead, GPT-2's technique for positional encoding parallels that of word embeddings. Each position within an input sequence is represented by a 1024-value one-hot vector. This positional representation goes through a linear layer to obtain a positional encoding.
• The Gaussian error linear unit (GELU) activation function is used in the feed-forward sublayers of GPT-2. GELU provides a blend of linear and non-linear activation properties that have been found to enhance model performance in deep learning tasks, particularly in NLPs and in training LLMs.
• We can build a GPT-2 model from scratch and load up the pre-trained weights released by OpenAI. The GPT-2 model you created can generate coherent text just as the original OpenAI GPT-2 model does.

FAQ