table of content

Part 1 Understanding attention and transformers

1 A tale of two models: Transformers and diffusions

1.1 What is a text-to-image generation model?

1.1.1 Unimodal vs. multimodal models

1.1.2 Practical use cases of text-to-image models

1.2 Transformer-based text-to-image generation

1.2.1 Converting an image into a sequence of integers and then back

1.2.2 Training and using a transformer-based text-to-image model

1.3 Text-to-image generation with diffusion models

1.3.1 Forward and reverse diffusions

1.3.2 Latent diffusion models and Stable Diffusion

1.4 How to build text-to-image models from scratch

1.5 Challenges for text-to-image generation models

1.5.1 Are generative AI models stealing from artists?

1.5.2 The geometric inconsistency problem

1.6 Social, environmental, and ethical concerns

2 Build a transformer

2.1 An overview of attention and transformers

2.1.1 How the attention mechanism works

2.1.2 How to create a transformer

2.2 Word embedding and positional encoding

2.2.1 Word tokenization with the Spacy library

2.2.2 A sequence padding function

2.2.3 Input embedding from word embedding and positional encoding

2.3 Creating an encoder–decoder transformer

2.3.1 Coding the attention mechanism

2.3.2 Defining the Transformer() class

2.3.3 Creating a language translator

2.4 Training and using the German-to-English translator

2.4.1 Training the encoder–decoder transformer

2.4.2 Translating German to English with the trained model

3 Classify images with a vision transformer

3.1 The blueprint to train a ViT

3.1.1 Converting images to sequences

3.1.2 Training a ViT for classification

3.2 The CIFAR-10 dataset

3.2.1 Downloading and visualizing CIFAR-10 images

3.2.2 Preparing datasets for training and testing

3.3 Building a ViT from scratch

3.3.1 Dividing images into patches

3.3.2 Modeling the positions of different patches in an image

3.3.3 Using the multi-head self-attention mechanism

3.3.4 Building an encoder-only transformer

3.3.5 Using the ViT to create a classifier

3.4 Training and using the ViT to classify images

3.4.1 Choosing the optimizer and the loss function

3.4.2 Training the ViT for image classification

3.4.3 Classifying images using the trained ViT

4 Add captions to images

4.1 Training and using a transformer to add captions

4.1.1 Preparing data and the causal attention mask

4.1.2 Creating and training a transformer

4.2 Preparing the training dataset

4.2.1 Downloading and visualizing Flickr 8k images

4.2.2 Building a vocabulary of tokens

4.2.3 Preparing the training dataset

4.3 Creating a multimodal transformer to add captions

4.3.1 Defining a ViT as the image encoder

4.3.2 Creating the decoder to generate text

4.4 Training and using the image-to-text transformer

4.4.1 Training the encoder–decoder transformer

4.4.2 Adding captions to images with the trained model

Part 2 Introduction to diffusion models

5 Generate images with diffusion models

5.1 The forward diffusion process

5.1.1 How diffusion models work

5.1.2 Visualizing the forward diffusion process

5.1.3 Different diffusion schedules

5.2 The reverse diffusion process

5.3 A blueprint to train the U-Net model

5.3.1 Steps in training a denoising U-Net model

5.3.2 Preprocessing the training data

5.4 Training and using the diffusion model

5.4.1 The Denoising Diffusion Probabilistic Model noise scheduler

5.4.2 Inference using the U-Net denoising model

5.4.3 Training and using the denoising U-Net model

6 Control what images to generate in diffusion models

6.1 Classifier-free guidance in diffusion models

6.1.1 An overview of classifier-free guidance

6.1.2 A blueprint to implement CFG

6.2 Different components of a denoising U-Net model

6.2.1 Time step embedding and label embedding

6.2.2 The U-Net denoising model architecture

6.2.3 Down blocks and up blocks in the U-Net

6.3 Building and training the denoising U-Net model

6.3.1 Building the denoising U-Net

6.3.2 The Denoising Diffusion Probabilistic Model

6.3.3 Training the diffusion model

6.4 Generating images with the trained diffusion model

6.4.1 Visualizing generated images

6.4.2 How the guidance parameter affects generated images

7 Generate high-resolution images with diffusion models

7.1 Attention in U-Net, DDIM, and image interpolation

7.1.1 Incorporating the attention mechanism in the U-Net model

7.1.2 Denoising Diffusion Implicit Models

7.1.3 Image interpolation in diffusion models

7.2 High-resolution flower images as training data

7.2.1 Visualizing images in the training dataset

7.2.2 Applying forward diffusion on flower images

7.3 Building and training a U-Net for high-resolution images

7.3.1 Building the denoising U-Net model

7.3.2 Training the denoising U-Net model

7.4 Image generation and interpolation

7.4.1 Using the trained denoising U-Net to generate images

7.4.2 Transition from one image to another

Part 3 Text-to-image generation with diffusion models

8 CLIP: A model to measure the similarity between image and text

8.1 The CLIP model

8.1.1 How the CLIP model works

8.1.2 Selecting an image from Flickr 8k based on a text description

8.2 Preparing the training dataset

8.2.1 Image-caption pairs in Flickr 8k

8.2.2 The DistilBERT tokenizer

8.2.3 Preprocess captions and images for training

8.3 Creating a CLIP model

8.3.1 Creating a text encoder

8.3.2 Creating an image encoder

8.3.3 Building a CLIP model

8.4 Training and using the CLIP model

8.4.1 Training the CLIP model

8.4.2 Using the trained CLIP model to select images

8.4.3 Using the OpenAI pretrained CLIP model to select images

9 Text-to-image generation with latent diffusion

9.1 What is a latent diffusion model?

9.1.1 How variational autoencoders work

9.1.2 Combining a latent diffusion model with a variational autoencoder

9.2 Compressing and reconstructing images with VAEs

9.2.1 Downloading the pretrained VAE

9.2.2 Encoding and decoding images with the pretrained VAE

9.3 Text-to-image generation with latent diffusion

9.3.1 Guidance by the CLIP model

9.3.2 Diffusion in the latent space

9.3.3 Converting latent images to high-resolution ones

9.4 Modifying existing images with text prompts

10 A deep dive into Stable Diffusion

10.1 Generating images with Stable Diffusion

10.2 The Stable Diffusion architecture

10.2.1 Generating images from text with Stable Diffusion

10.2.2 Text embedding interpolation

10.3 Creating text embeddings

10.4 Image generation in the latent space

10.5 Converting latent images to high-resolution ones

Part 4 Text-to-image generation with transformers

11 VQGAN: Convert images into sequences of integers

11.1 Converting images into sequences of integers and back

11.2 Variational autoencoders

11.2.1 What is an autoencoder?

11.2.2 The need for VAEs and their training methodology

11.3 Vector quantized variational autoencoders

11.3.1 The need for VQ-VAEs

11.3.2 The VQ-VAE model architecture and training process

11.4 Vector quantized generative adversarial networks

11.4.1 Generative adversarial networks

11.4.2 VQGAN: A GAN with a VQ-VAE generator

11.5 A pretrained VQGAN model

11.5.1 Reconstructing images with the pretrained VQGAN

11.5.2 Converting images into sequences of integers

12 A minimal implementation of DALL-E

12.1 How min-DALL-E works

12.1.1 Training min-DALL-E

12.1.2 From prompt to pixels: Image generation at inference time

12.2 Tokenizing and encoding the text prompt

12.2.1 Tokenizing the text prompt

12.2.2 Encoding the text prompt

12.3 Iterative prediction of image tokens

12.3.1 Loading the pretrained BART decoder

12.3.2 Predicting image tokens using the BART decoder

12.4 Converting image tokens to high-resolution images

12.4.1 Loading the pretrained VQGAN detokenizer

12.4.2 Visualizing the intermediate and final high-resolution outputs

Part 5 New developments and challenges

13 New developments and challenges in text-to-image generation

13.1 State-of-the-art text-to-image generators

13.1.1 DALL-E series

13.1.2 Google’s Imagen

13.1.3 Latent diffusion models: Stable Diffusion and Midjourney

13.2 Challenges and concerns

13.3 A blueprint to fine-tune ResNet50

13.3.1 The history and architecture of ResNet50

13.3.2 A plan to fine-tune ResNet50 for classification

13.3.3 Using ResNet50 to classify images

13.4 Fine-tuning ResNet50 to detect fake images

13.4.1 Downloading and preprocessing real and fake face images

13.4.2 Fine-tuning ResNet50

13.4.3 Detecting deepfakes using the fine-tuned ResNet50

Appendix

Installing PyTorch and enabling GPU training locally and in Colab

A.1 Installing Python and setting up a virtual environment

A.1.1 Installing Anaconda

A.1.2 Setting up a Python virtual environment

A.1.3 Installing Jupyter Notebook

A.2 Installing PyTorch

A.3 Using Google Colab for GPU training and inference

references

Overview

4 Add captions to images

This chapter focuses on building the reverse of text-to-image generation—an image-to-text captioning system—as a practical stepping stone to multimodal generative models. It explains how both directions share the same core idea: mapping between visual and linguistic representations through an encoder-decoder transformer. By implementing a captioning model from scratch, you learn the mechanics of cross-modal attention and autoregressive decoding in a setting that is easier to train and reason about, while laying the groundwork for more complex text-to-image systems later on.

The model pairs a vision transformer (encoder) with a text generator (decoder). Images are standardized and split into patches that are turned into embeddings with positional information, producing a sequence representation of the picture. Captions are tokenized, wrapped with special start and end tokens, and transformed into input–target pairs by shifting the sequence to train next-token prediction. A causal attention mask in the decoder enforces that each token only attends to previous tokens, while cross-attention aligns the evolving text with the image embeddings. The chapter also covers batching, padding, vocabulary construction, and the overall data pipeline using image–caption pairs such as those in Flickr 8k.

Training minimizes cross-entropy between the predicted token distribution and the shifted target sequence, teaching the decoder to generate captions conditioned on the image and previously generated tokens. At inference, the model encodes the image once and decodes tokens autoregressively from a start symbol until an end symbol, with temperature controlling the diversity of the samples. The resulting captions are generally coherent and faithful to image content, demonstrating the effectiveness of the encoder–decoder design and preparing you for subsequent chapters that build on these ideas for text-to-image generation, including diffusion-based approaches.

Eight steps to build a text-to-image generator from scratch. In this chapter, you will reinforce and apply skills from steps 1 and 2: building a transformer that captions images. The encoder is a vision transformer (ViT), similar to what you created for image classification in chapter 3. The decoder generates text based on image features, much like the decoder you developed for German-to-English translation in chapter 2.

How to prepare data to train a transformer to add captions to images. We first obtain a training dataset of images paired with descriptive captions. The images are standardized to a certain size (for example, 3x128x128) so that we can feed them to the image encoder in the transformer. The captions are first tokenized, and we add <start> and <end> to the beginning and end of the caption. We omit the <end> token and use the rest of the tokens as the input sequence ([<start>, two, bears, fight]). We slide the input sequence one token to the right and use the result as the target sequence ([two, bears, fight, <end>]). The input sequence is fed to the decoder to make predictions on the next token. The target sequence serves as the ground truth. We compare the predicted output from the decoder with the target sequence and calculate the cross-entropy loss. During training, we modify model parameters to minimize this loss. This approach compels the model to predict the next token based on previous ones.

How the causal attention mask works. To force the model to predict the next token based on all tokens before it, but not tokens after it (future tokens), we apply the causal attention mask when calculating self-attention in the transformer’s decoder layer. Specifically, we mask out the attention weights above the main diagonal so that the model cannot access future tokens. The left matrix shows the attention weights before the causal attention mask and the right matrix shows the attention weights after we apply the causal attention mask. As a result, in the first iteration, the model predicts the token “two” using only the token “<start>”. In the second iteration, the model predicts the token “bears” using only tokens “<start> two” but not future tokens “bears fight.” In the last iteration, the model uses all four tokens “<start> two bears fight” to predict the next token (which is the token <end>).

The structure of an encoder-decoder transformer to add captions to images. The encoder in the transformer (left side of the diagram), which consists of N identical encoder layers, encodes an image into image embeddings that capture the features of the image. It then passes this tensor to the decoder (right side of the diagram), which consists of N identical decoder layers. The decoder takes the input sequence as the input ([<start>, two, bears, fight]) and predicts the output sequence. We compare the predictions with the ground truth, which is [two, bears, fight, <end>]. We modify the model parameters to minimize the cross-entropy loss. Since a causal attention mask is applied in each decoder layer, this is equivalent to training the model to predict the next token based on all tokens before it (but not tokens after it) in the sequence.

Adding a caption to an image using the trained transformer. We first feed the image to the trained encoder to obtain the image embeddings. We then pass the image embeddings to the decoder to generate a caption. The decoder constructs the caption in an autoregressive fashion. It starts with the beginning of the caption token, <start>, and predicts the next token in the caption. It then adds the prediction to the caption and predicts the token after it. It repeats the process until the predicted token is <end>, which signals the end of the caption.

Comparing generated captions with original captions. In this figure, we present three examples of generated captions with the trained encoder-decoder transformer. We display both the generated captions and the original captions on top of each image.

Summary

Training a multi-modal transformer for text-to-image generation (i.e., adding captions to images) and training one for image-to-text generation (i.e., generating images based on textual descriptions) share some similarities. Both tasks involve learning complex mappings between textual and visual modalities.
Understanding and training multi-modal transformers for image-to-text generation first can provide valuable insights and foundational knowledge, making the complex task of text-to-image generation more manageable.
We create an encoder-decoder transformer for image-to-text generation. The encoder converts an image into a sequence of image patches. It then uses image embedding to convert image patches to vector representations that capture the feature maps in these patches. The decoder generates captions one token at a time, based on the output from the encoder and previously generated tokens in the caption.
When preparing the training data, we generate input sequences and target sequences based on captions associated with the training images. The decoder tries to predict the next token after the input sequence.

For example, if the caption associated with an image is “a dog runs”, we’ll add <start> and <end> to the beginning and the end of the caption. We’ll use “<start> a dog runs” as the input sequence. We move the input sequence one token to the right and use it as the target sequence (i.e., the ground truth).
During training, we feed the input sequence to the model and compare the model output with the target sequence, “a dog runs <end>,” and calculate the cross-entropy loss. We modify model parameters to minimize this loss.

FAQ

Why start with image captioning before building a text-to-image model?

Image captioning is simpler yet teaches the key multimodal skills you’ll reuse for text-to-image: extracting visual features with a Vision Transformer (ViT), aligning image and text via cross-attention, and generating sequences autoregressively with a decoder. Mastering these makes the later, more complex task of synthesizing images from text more approachable.

What architecture is used to add captions to images?

An encoder–decoder transformer. The encoder is a ViT that converts an image into a sequence of patch embeddings with positional encodings. The decoder is a text generator that, using causal self-attention and cross-attention to the image embeddings, produces a caption token by token from a special <start> token until <end>.

How do input and target sequences for training get constructed?

For a caption like “two bears fight,” you add <start> and <end>. The input sequence is [<start>, two, bears, fight] and the target sequence is [two, bears, fight, <end>]. The model learns to predict the next token at each position using cross-entropy loss (teacher forcing) while masking future tokens.

What is a causal attention mask and why is it needed?

The causal mask prevents the decoder from attending to future tokens during self-attention by zeroing (or -∞ masking) attention weights above the main diagonal. This enforces left-to-right generation so each prediction depends only on previously known tokens, not future ones.

Which dataset and preprocessing steps are used?

Flickr8k (≈8k images, 5 captions each). Images are resized to 3×128×128. Captions are tokenized; <start> and <end> are added. A vocabulary is built with special tokens <pad>, <start>, <end>, <unk>. Sequences are mapped to indices and padded per batch; DataLoader batches (e.g., size 128) are created for training and testing.

How does the Vision Transformer encode images here?

Each 128×128 image is split into 8×8 patches (16×16 grid = 256 patches). Each patch (3×8×8 = 192 values) is linearly projected to hidden_size, positional embeddings are added, and the sequence passes through multiple transformer blocks (e.g., 6) with multi-head self-attention and feed-forward layers.

What are the key hyperparameters and their roles?

- hidden_size (e.g., 192): dimensionality of patch/token embeddings. - num_layers (encoder, decoder), e.g., (6, 6): model depth and capacity. - num_heads (e.g., 8): parallel attention subspaces. - patch_size (e.g., 8): controls number of patches/sequence length. - Batch size, learning rate (e.g., 1e-4): training stability/speed. This setup yields ~9.5M parameters in the example model.

How is the model trained and what loss is used?

The model is trained to predict the next token at every position with a cross-entropy loss over the vocabulary. Padding masks exclude padded positions from the loss. Adam optimizer (e.g., lr=1e-4) and mixed precision (torch.amp) accelerate training. Training proceeds over epochs, saving the model state_dict afterward.

How does caption generation (inference) work?

At test time, encode the image with the trained encoder. Start the decoder with <start> and iteratively sample the next token (up to a max length or until <end>), conditioning on the image embeddings and previously generated tokens. A temperature parameter controls randomness: <1 is more deterministic; >1 is more diverse.

How does cross-attention align text with image features?

In each decoder layer, queries (Q) come from the current text embeddings, while keys (K) and values (V) come from the encoder’s image embeddings. The attention weights QKᵀ/√dₖ highlight image patches relevant to the current decoding step, and the weighted sum over V feeds information about those patches into the decoder to guide the next-token prediction.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more