Overview
2 Build a transformer
This chapter establishes attention and transformers as the core tools you’ll use throughout the book to build a text-to-image generator from scratch. It motivates their importance by noting that transformer-based models can generate images as sequences of patches and that even non-transformer systems like diffusion models rely on attention and multimodal encoders to align images with text. The chapter’s objective is to demystify attention, then guide you through implementing and training an encoder–decoder transformer on a real translation task so you gain the intuition and coding skills needed for later, more advanced generative systems.
The chapter explains scaled dot-product attention with query, key, and value, showing how softmax-normalized similarity scores produce context-aware token representations. It walks through tokenization, word embeddings, and sinusoidal positional encodings that inject order into parallel processing. Building on this, it introduces multi-head attention to capture diverse relationships, and details the encoder–decoder architecture: stacked encoder layers of self-attention plus position-wise feed-forward networks with residual connections and layer normalization; and stacked decoder layers that add masked self-attention for autoregressive prediction and cross-attention to the encoder output. It also contrasts encoder-only, decoder-only, and encoder–decoder variants and highlights attention’s role in resolving contextual ambiguity.
On the practical side, the chapter covers preparing a dataset of sentence pairs, tokenizing both languages, building vocabularies, and using special tokens (BOS, EOS, PAD, UNK). It shows how to map tokens to indices, sort, batch, and pad sequences; implement embeddings and positional encodings; code attention, multi-head attention, and the Transformer wrapper; and train with an optimizer schedule, cross-entropy loss, and label smoothing. Finally, it demonstrates inference via iterative next-token prediction from BOS until EOS, producing translations that closely match references and setting the stage for applying transformers to vision tasks and, ultimately, building a complete text-to-image system.
The eight-step roadmap for building a text-to-image generator from scratch. This chapter launches your journey by guiding you through the construction and training of an encoder-decoder transformer for German-to-English translation, giving you hands-on experience with attention mechanisms and transformer architectures, the essential building blocks for all subsequent steps.
How to convert raw text into input embeddings that we can feed into the transformer model. A German phrase (for example, "Wei geht es dir," which means "How are you?") is broken down into a sequence of smaller semantic units known as tokens ([Wie, geht, es, dir]). Each token is then transformed into a word embedding that captures its meaning. Additionally, positional encodings are created to retain the order of tokens within the sequence. Finally, the word embeddings and positional encodings are combined element-wise to produce the final input embedding. We perform the same procedures to create word embeddings, positional encodings, and eventually input embedding for English phrases.
A diagram of the self-attention mechanism. To calculate attention, the input embedding X is first passed through three neural layers with weights, WQ, WK, and WV, respectively. The outputs are query Q, key K, and value V. The scaled attention score is the product of Q and K divided by the square root of the dimension of K, dk. We apply the softmax function (which we’ll explain shortly) on the scaled attention score to obtain the attention weight. The attention is the product of the attention weight and value V.
An encoder-decoder transformer for machine language translation (e.g., German to English). The encoder in the transformer (left side of the diagram), which consists of N identical encoder layers, learns the meaning of the input sequence and converts it into a tensor that represents this meaning. It then passes this tensor to the decoder (right side of the diagram), which consists of N identical decoder layers. The decoder constructs the output (e.g., the English translation of a German phrase) by predicting one token at a time, based on previous tokens in the sequence and the output from the encoder. The generator on the top right is the head attached to the output from the decoder so that the output is the probability distribution over all tokens in the target language (e.g., the English vocabulary).
The structure of the encoder in the transformer. The encoder consists of N = 6 identical encoder layers. Each encoder layer contains two sublayers. The first is a multi-head self-attention layer and the second is a feed-forward network. Each sublayer uses layer normalization and residual connection. The input to the encoder is the embedding of the German phrase (e.g., “Wie geht es dir”) and the output is an abstract representation that captures the meaning of the phrase.
The structure of the decoder in the transformer. The decoder consists of N = 6 identical decoder layers. Each decoder layer contains three sublayers. The first is a masked multi-head self-attention layer. The second is a multi-head cross-attention layer to calculate the cross attention between the output from the first sublayer and the output from the encoder. The third sublayer is a feed-forward network. Each sublayer uses layer normalization and residual connection.
Summary
- Transformers are advanced deep-learning models that excel in handling sequence-to-sequence prediction challenges. Their strength lies in effectively understanding the relationships between elements in input and output sequences over long distances.
- The key ingredient of the transformer architecture is its attention mechanism. This mechanism assesses the relationship between words in a sequence by assigning weights, determining how closely words are related based on the training data. This enables transformer models like ChatGPT to comprehend relationships between words, thus understanding human language more effectively.
- A common way to calculate attention is scaled dot product attention (SDPA). This method is also called self-attention because the algorithm calculates how a word attends to all words in the sequence, including the word itself.
- Furthermore, instead of using one set of query, key, and value vectors, transformer models use a concept called multi-head attention. The query, key, and value vectors are split into multiple heads. Each head pays attention to different parts or aspects of the input, enabling the model to capture a broader range of information and form a more detailed and contextual understanding of the input data.
- Word embeddings are vector representations of tokens used in natural language processing (NLP) tasks. They map tokens into continuous vector spaces, allowing the model to process and understand their meanings numerically.
- Transformers lack inherent sequential understanding, as they process tokens in parallel rather than sequentially. Positional encoding introduces information about the token order into the model.
FAQ
Why does the book start with transformers when the goal is text-to-image generation?
Because attention and transformers underpin most modern generative models. Autoregressive image generators (like DALL·E) predict image patches as a sequence using transformers, and diffusion models rely on attention inside U-Nets and use a multimodal transformer (e.g., CLIP) to align text and images. Mastering attention and a basic encoder-decoder transformer builds the skills needed for both pathways.How do query, key, and value work in scaled dot-product attention (SDPA)?
Given input embeddings X, the model learns three linear projections: Q = X·WQ, K = X·WK, V = X·WV. Attention weights are computed as softmax(Q·Kᵀ / √dₖ), where dₖ is the key dimension. The final output is Attention(Q, K, V) = softmax(Q·Kᵀ / √dₖ)·V. This lets each token reweight information from all tokens (including itself) by learned similarity.What is multi-head attention and why split into multiple heads?
Multi-head attention partitions the Q/K/V feature dimensions into several heads (e.g., d_model=256 split into 8 heads of 32). Each head learns different relationships (syntax, long-range links, disambiguation), and their outputs are combined. This parallel, diversified focus yields richer, more robust context than a single-head mechanism.What are word embeddings and positional encodings, and how are they combined?
- Word embeddings map token indices to dense vectors (e.g., 256 dims) learned during training, capturing semantic similarity.- Positional encodings inject order information using fixed sine/cosine patterns (even indices use sin, odd use cos).
- The model adds positional encodings to word embeddings to form input embeddings that carry both meaning and position.
How is the encoder-decoder transformer organized for translation?
- Encoder: N identical layers (e.g., 6), each with multi-head self-attention + position-wise feed-forward, plus layer normalization and residual connections. It encodes the source sentence into a “memory” tensor.- Decoder: N identical layers with masked self-attention, cross-attention over the encoder’s memory (query = decoder states; key/value = encoder output), and a feed-forward network. A generator head maps outputs to a probability distribution over the target vocabulary at each step.
Why is masking needed in the decoder’s self-attention?
Masking hides future positions so each prediction only depends on previously generated tokens. This aligns training with inference (autoregressive generation) and prevents the model from “cheating” by peeking ahead.How are the German–English datasets tokenized and indexed?
Using spaCy tokenizers for German and English, sentences are split into tokens. Special tokens are added: BOS (begin), EOS (end), PAD (padding), UNK (unknown). Vocabularies map tokens↔indices for both languages. The chapter uses ~29,000 parallel sentence pairs to build these vocabularies.How are variable-length sequences batched and padded efficiently?
Sequences in a batch are padded with PAD (index 0) to the length of the longest sequence. Sorting by length before batching groups similar-length sentences together, minimizing padding and speeding up training. Masks (e.g., src_mask) hide padding during attention.How is the model instantiated and trained?
The create_model function assembles encoder, decoder, embeddings (with positional encodings), and a generator. Example hyperparameters: N=6 layers, d_model=256, d_ff=1024, h=8 heads, dropout=0.1. Training uses Adam with a Noam learning-rate schedule and cross-entropy loss (optionally with label smoothing), over multiple epochs, saving learned weights on completion.How does inference (German→English) work step by step?
- Tokenize the German input, add BOS/EOS, and encode it into memory using the encoder.- Start the decoder with BOS in the target language.
- Repeatedly: apply masked decoding, project to vocabulary with the generator, pick the next token (e.g., greedy argmax), append it, and continue until EOS or a max length is reached.
- Detokenize to form the final English sentence.
Build a Text-to-Image Generator (from Scratch) ebook for free
