table of content

Part 1 Understanding attention and transformers

1 A tale of two models: Transformers and diffusions

1.1 What is a text-to-image generation model?

1.1.1 Unimodal vs. multimodal models

1.1.2 Practical use cases of text-to-image models

1.2 Transformer-based text-to-image generation

1.2.1 Converting an image into a sequence of integers and then back

1.2.2 Training and using a transformer-based text-to-image model

1.3 Text-to-image generation with diffusion models

1.3.1 Forward and reverse diffusions

1.3.2 Latent diffusion models and Stable Diffusion

1.4 How to build text-to-image models from scratch

1.5 Challenges for text-to-image generation models

1.5.1 Are generative AI models stealing from artists?

1.5.2 The geometric inconsistency problem

1.6 Social, environmental, and ethical concerns

2 Build a transformer

2.1 An overview of attention and transformers

2.1.1 How the attention mechanism works

2.1.2 How to create a transformer

2.2 Word embedding and positional encoding

2.2.1 Word tokenization with the Spacy library

2.2.2 A sequence padding function

2.2.3 Input embedding from word embedding and positional encoding

2.3 Creating an encoder–decoder transformer

2.3.1 Coding the attention mechanism

2.3.2 Defining the Transformer() class

2.3.3 Creating a language translator

2.4 Training and using the German-to-English translator

2.4.1 Training the encoder–decoder transformer

2.4.2 Translating German to English with the trained model

3 Classify images with a vision transformer

3.1 The blueprint to train a ViT

3.1.1 Converting images to sequences

3.1.2 Training a ViT for classification

3.2 The CIFAR-10 dataset

3.2.1 Downloading and visualizing CIFAR-10 images

3.2.2 Preparing datasets for training and testing

3.3 Building a ViT from scratch

3.3.1 Dividing images into patches

3.3.2 Modeling the positions of different patches in an image

3.3.3 Using the multi-head self-attention mechanism

3.3.4 Building an encoder-only transformer

3.3.5 Using the ViT to create a classifier

3.4 Training and using the ViT to classify images

3.4.1 Choosing the optimizer and the loss function

3.4.2 Training the ViT for image classification

3.4.3 Classifying images using the trained ViT

4 Add captions to images

4.1 Training and using a transformer to add captions

4.1.1 Preparing data and the causal attention mask

4.1.2 Creating and training a transformer

4.2 Preparing the training dataset

4.2.1 Downloading and visualizing Flickr 8k images

4.2.2 Building a vocabulary of tokens

4.2.3 Preparing the training dataset

4.3 Creating a multimodal transformer to add captions

4.3.1 Defining a ViT as the image encoder

4.3.2 Creating the decoder to generate text

4.4 Training and using the image-to-text transformer

4.4.1 Training the encoder–decoder transformer

4.4.2 Adding captions to images with the trained model

Part 2 Introduction to diffusion models

5 Generate images with diffusion models

5.1 The forward diffusion process

5.1.1 How diffusion models work

5.1.2 Visualizing the forward diffusion process

5.1.3 Different diffusion schedules

5.2 The reverse diffusion process

5.3 A blueprint to train the U-Net model

5.3.1 Steps in training a denoising U-Net model

5.3.2 Preprocessing the training data

5.4 Training and using the diffusion model

5.4.1 The Denoising Diffusion Probabilistic Model noise scheduler

5.4.2 Inference using the U-Net denoising model

5.4.3 Training and using the denoising U-Net model

6 Control what images to generate in diffusion models

6.1 Classifier-free guidance in diffusion models

6.1.1 An overview of classifier-free guidance

6.1.2 A blueprint to implement CFG

6.2 Different components of a denoising U-Net model

6.2.1 Time step embedding and label embedding

6.2.2 The U-Net denoising model architecture

6.2.3 Down blocks and up blocks in the U-Net

6.3 Building and training the denoising U-Net model

6.3.1 Building the denoising U-Net

6.3.2 The Denoising Diffusion Probabilistic Model

6.3.3 Training the diffusion model

6.4 Generating images with the trained diffusion model

6.4.1 Visualizing generated images

6.4.2 How the guidance parameter affects generated images

7 Generate high-resolution images with diffusion models

7.1 Attention in U-Net, DDIM, and image interpolation

7.1.1 Incorporating the attention mechanism in the U-Net model

7.1.2 Denoising Diffusion Implicit Models

7.1.3 Image interpolation in diffusion models

7.2 High-resolution flower images as training data

7.2.1 Visualizing images in the training dataset

7.2.2 Applying forward diffusion on flower images

7.3 Building and training a U-Net for high-resolution images

7.3.1 Building the denoising U-Net model

7.3.2 Training the denoising U-Net model

7.4 Image generation and interpolation

7.4.1 Using the trained denoising U-Net to generate images

7.4.2 Transition from one image to another

Part 3 Text-to-image generation with diffusion models

8 CLIP: A model to measure the similarity between image and text

8.1 The CLIP model

8.1.1 How the CLIP model works

8.1.2 Selecting an image from Flickr 8k based on a text description

8.2 Preparing the training dataset

8.2.1 Image-caption pairs in Flickr 8k

8.2.2 The DistilBERT tokenizer

8.2.3 Preprocess captions and images for training

8.3 Creating a CLIP model

8.3.1 Creating a text encoder

8.3.2 Creating an image encoder

8.3.3 Building a CLIP model

8.4 Training and using the CLIP model

8.4.1 Training the CLIP model

8.4.2 Using the trained CLIP model to select images

8.4.3 Using the OpenAI pretrained CLIP model to select images

9 Text-to-image generation with latent diffusion

9.1 What is a latent diffusion model?

9.1.1 How variational autoencoders work

9.1.2 Combining a latent diffusion model with a variational autoencoder

9.2 Compressing and reconstructing images with VAEs

9.2.1 Downloading the pretrained VAE

9.2.2 Encoding and decoding images with the pretrained VAE

9.3 Text-to-image generation with latent diffusion

9.3.1 Guidance by the CLIP model

9.3.2 Diffusion in the latent space

9.3.3 Converting latent images to high-resolution ones

9.4 Modifying existing images with text prompts

10 A deep dive into Stable Diffusion

10.1 Generating images with Stable Diffusion

10.2 The Stable Diffusion architecture

10.2.1 Generating images from text with Stable Diffusion

10.2.2 Text embedding interpolation

10.3 Creating text embeddings

10.4 Image generation in the latent space

10.5 Converting latent images to high-resolution ones

Part 4 Text-to-image generation with transformers

11 VQGAN: Convert images into sequences of integers

11.1 Converting images into sequences of integers and back

11.2 Variational autoencoders

11.2.1 What is an autoencoder?

11.2.2 The need for VAEs and their training methodology

11.3 Vector quantized variational autoencoders

11.3.1 The need for VQ-VAEs

11.3.2 The VQ-VAE model architecture and training process

11.4 Vector quantized generative adversarial networks

11.4.1 Generative adversarial networks

11.4.2 VQGAN: A GAN with a VQ-VAE generator

11.5 A pretrained VQGAN model

11.5.1 Reconstructing images with the pretrained VQGAN

11.5.2 Converting images into sequences of integers

12 A minimal implementation of DALL-E

12.1 How min-DALL-E works

12.1.1 Training min-DALL-E

12.1.2 From prompt to pixels: Image generation at inference time

12.2 Tokenizing and encoding the text prompt

12.2.1 Tokenizing the text prompt

12.2.2 Encoding the text prompt

12.3 Iterative prediction of image tokens

12.3.1 Loading the pretrained BART decoder

12.3.2 Predicting image tokens using the BART decoder

12.4 Converting image tokens to high-resolution images

12.4.1 Loading the pretrained VQGAN detokenizer

12.4.2 Visualizing the intermediate and final high-resolution outputs

Part 5 New developments and challenges

13 New developments and challenges in text-to-image generation

13.1 State-of-the-art text-to-image generators

13.1.1 DALL-E series

13.1.2 Google’s Imagen

13.1.3 Latent diffusion models: Stable Diffusion and Midjourney

13.2 Challenges and concerns

13.3 A blueprint to fine-tune ResNet50

13.3.1 The history and architecture of ResNet50

13.3.2 A plan to fine-tune ResNet50 for classification

13.3.3 Using ResNet50 to classify images

13.4 Fine-tuning ResNet50 to detect fake images

13.4.1 Downloading and preprocessing real and fake face images

13.4.2 Fine-tuning ResNet50

13.4.3 Detecting deepfakes using the fine-tuned ResNet50

Appendix

Installing PyTorch and enabling GPU training locally and in Colab

A.1 Installing Python and setting up a virtual environment

A.1.1 Installing Anaconda

A.1.2 Setting up a Python virtual environment

A.1.3 Installing Jupyter Notebook

A.2 Installing PyTorch

A.3 Using Google Colab for GPU training and inference

references

Overview

5 Generate images with diffusion models

This chapter introduces diffusion models as the current standard for high-quality image generation and uses a practical, build-from-scratch approach to make the concepts concrete. It explains the two complementary phases at the heart of diffusion: a forward process that gradually corrupts images with noise and a reverse process that learns to remove that noise step by step to synthesize new samples. The chapter positions these ideas as foundational for modern text-to-image systems and guides the reader toward hands-on implementation by generating grayscale clothing images, laying groundwork for conditional diffusion covered later.

The core mechanics are developed by first visualizing forward diffusion under different noise schedules and then learning how to invert it. Linear and cosine schedules are compared to show how the rate of noise addition influences learning and output quality, with the cosine schedule typically producing more stable training and better results. The reverse process is framed as learning to predict the injected noise at each timestep; a U-Net serves as the denoiser, taking a noisy image and its timestep as input and outputting the noise estimate. With this, the model can iteratively refine pure noise into a clean image.

Implementation centers on training a denoising U-Net with Fashion MNIST: images are resized and normalized, random timesteps are sampled per batch, and noise is added via a DDPM scheduler. The model is optimized to minimize the L1 difference between predicted and true noise using AdamW and mixed precision to improve speed. During and after training, sampling begins from random noise and proceeds through the scheduler’s timesteps, repeatedly applying the U-Net to remove noise and form recognizable clothing items. The results improve across epochs, demonstrating how the learned denoiser and scheduler together enable image synthesis and setting the stage for adding conditioning signals to control what the model generates.

The eight-step roadmap for building a text-to-image generator from scratch. This chapter is dedicated to step 3: diving deep into the mechanics of forward and reverse diffusion, and developing a diffusion model capable of generating images from noise.

A diagram of the forward diffusion process. We start with a clean image (at time step 0) from the training set and add noise to it to form a noisy image (at time step 1). In the next time step, we inject more noise into the image. We repeat this process 1000 time steps until the image becomes random noise.

The forward diffusion process. The four images in the first column are the original clean images. We then gradually add noise to these images from time step 1 to time step 1000. As the time step increases, more and more noise is injected into the images. The four images in the second column are images after 200 time steps. The third column contains images after 400 time steps and they have more noise than those in the second column. The last column contains images after 1000 time steps and they are 100% random noise.

The weight placed on noise under linear and cosine diffusion schedules. The x-axis is the relative time step, t/T, which ranges from 0 to 1. The y-axis is the weight placed on the noise in the forward diffusion process. The solid line represents the linear diffusion schedule while the dotted line represents the cosine diffusion schedule. The weight on the noise under the cosine diffusion schedule increases over time more gradually compared to that under the linear diffusion schedule.

The weight placed on the clean image under linear and cosine diffusion schedules. The x-axis is the relative time step, t/T, which ranges from 0 to 1. The y-axis is the weight placed on the clean image in the forward diffusion process. The solid line represents the linear diffusion schedule while the dotted line represents the cosine diffusion schedule. The weight on the clean image under the cosine diffusion schedule decreases over time more gradually compared to that under the linear diffusion schedule.

How noise is gradually added to images in the forward diffusion process under linear and cosine diffusion schedules. We gradually add noise to a clean image under a linear diffusion schedule (top row) and a cosine diffusion schedule (bottom row). The first column contains clean images (i.e., t=0). As we move to the right, the time steps are 166, 332, 498, 664, 830, and 996, respectively. The linear diffusion schedule adds noise to the image more quickly, while the cosine diffusion schedule adds noise more gradually over time.

How the reverse diffusion process works. Upon observing xt+1, we cannot obtain xt because the function q(xt|xt+1) is unknown. Therefore, we need to use the training data to learn a model to approximate the conditional probability pθ(xt|xt+1), where θ are model parameters.

Steps to train a denoising U-Net model to remove noise from images. We first obtain clean clothing-item images as our training set. We add noise to clean images and present them to the U-Net model. The model predicts the noise in the noisy images based on the time step the image is in. We compare the predicted noise with the actual noise injected into the image and adjust the model weights to minimize the mean absolute error.

Grayscale images of clothing items in the Fashion MNIST dataset. There are ten different types of clothing items, labeled from 0 to 9. The ten text labels corresponding to the numerical labels 0 to 9 are 't-shirt', 'trouser', 'pullover', 'dress', 'coat', 'sandal', 'shirt', 'sneaker', 'bag', and 'ankle boot', respectively. The text label is on top of each image in this figure.

Clothing-item images generated by the denoising U-Net model after 1, 3. 5. 7, and 9 epochs of training. We train the model for 10 epochs. After each epoch of training, we use the model to generate clothing-item images. The top row shows the images after 1 epoch of training, the second shows those after 3 epochs of training, and so on. The bottom row shows the images after 9 epochs of training. As you can see, the quality of the generated images improves as the training progresses.

Generated clothing-item images with a trained denoising U-Net model. We train the model for ten epochs. We feed random noise to the trained model. The model iteratively removes noise from the noisy images for 1000 time steps and the result is shown in this figure.

Summary

In the forward diffusion process, we start with a sample of clean images. We gradually introduce small amounts of random noise into these clean images. After many time steps of adding noise, the clean images become pure random noise.
Two commonly used diffusion schedules in the forward process are the linear and cosine diffusion schedules.

A linear schedule defines the variance of noise added in each time step as an increasing linear function of time. The linear noise schedule is simple and commonly used due to ease of implementation.
The cosine schedule adjusts the noise variance in each time step using a cosine function. This leads to a non-linear progression where noise increases more gradually at first and then accelerates. The cosine schedule helps maintain finer details in initial diffusion steps, which can enhance model performance and output quality.

In the reverse diffusion process (also called the denoising process), we train a model to reverse the forward diffusion process. We feed the model with random noise and ask it to produce a noisy image. We iteratively repeat the process for many time steps and the end result is a clean image.
We usually use a U-Net as the denoising model. The U-Net has a symmetric shape with a contracting encoder path and an expansive decoder path, connected by a bottleneck layer. In denoising, U-Nets are adapted to remove noise while preserving details. U-Nets outperform simple convolutional networks in denoising tasks due to their efficient capturing of local and global features in images.

FAQ

What is a diffusion model and why is it well-suited for image generation?

Diffusion models generate images via two complementary processes: they learn to add small amounts of Gaussian noise to clean images (forward diffusion), and then learn to remove that noise step by step (reverse diffusion). Because denoising is broken into many small, learnable steps, these models tend to be stable and produce high-quality, high-resolution images that rival or surpass other generative approaches.

What happens during the forward diffusion process?

Starting from a clean image, the process injects a small amount of noise at each timestep until the image becomes nearly pure noise. The amount of noise is controlled by a schedule over T steps (often T=1000). Using cumulative weights (often written in terms of alpha and alpha-bar), you can directly sample a noisy image at any timestep by mixing the original image and Gaussian noise with timestep-dependent weights.

How does the reverse diffusion (denoising) process reconstruct images?

Since the exact reverse transition q(xₜ₋₁ | xₜ) is unknown, a neural network is trained to approximate it. In practice, the model predicts the noise present in xₜ given the timestep t. Subtracting the predicted noise (with the schedule’s weights) lets you iteratively refine xₜ into xₜ₋₁, and repeating this from pure noise produces a clean sample.

What is a noise schedule, and how do linear and cosine schedules differ?

A noise schedule specifies how much noise is added at each timestep in forward diffusion (and implicitly how denoising is structured). A linear schedule increases noise roughly linearly over time and is simple to implement. A cosine schedule increases noise more gradually and uniformly across timesteps, which often improves training dynamics and final sample quality. The trade-off is simplicity (linear) versus performance (cosine).

Why use a U-Net for denoising, and what do skip connections and attention add?

U-Nets combine an encoder (captures global context) and a decoder (reconstructs details) with skip connections that pass high-resolution features directly across the network. This helps preserve fine details while denoising. Attention blocks (when used) let the model focus on informative regions, further improving image fidelity and structure.

Which dataset and preprocessing steps are used in this chapter?

The chapter uses Fashion MNIST (10 clothing categories, grayscale). Images are resized from 28×28 to 32×32 to match the chosen U-Net configuration, converted to tensors, and normalized to the range [-1, 1] for stable training. Data are batched (e.g., batch size 128) and shuffled for efficient GPU/CPU processing.

How is the denoising U-Net trained (loss, optimizer, timesteps, precision)?

The model receives a batch of clean images, random timesteps per image, and injected Gaussian noise to create noisy inputs. It predicts the noise and is trained with L1 loss (mean absolute error) between predicted and actual noise. AdamW with a learning rate around 2e-4 is used, and automatic mixed precision (torch.amp) speeds up training. Typical training in the chapter runs for about 10 epochs.

What is the DDPM scheduler used here, and why 1,000 timesteps?

The chapter uses diffusers’ DDPMScheduler with 1,000 training timesteps and a cosine-based schedule variant (squaredcos_cap_v2). Using ~1,000 steps balances image quality and compute: fewer steps can degrade quality, while many more steps bring diminishing returns with higher cost. The scheduler provides add_noise during training and step during sampling.

How does the sampling function generate images from pure noise?

Sampling starts from random Gaussian noise of the target shape. For each timestep (set by the scheduler), the U-Net predicts the noise component in the current image; the scheduler’s step then computes the previous image in the chain. Repeating this for all timesteps yields a clean sample. Gradients are disabled during sampling (e.g., with torch.no_grad), and setting a random seed makes results reproducible.

How can I monitor training progress and troubleshoot poor samples?

Periodically generate and save sample grids after each epoch to visually track improvements. If samples look washed out or noisy: verify input normalization to [-1, 1]; ensure the scheduler used for training matches sampling; check that timesteps are sampled uniformly; confirm the loss decreases; and consider training longer, tweaking the learning rate, or using the cosine schedule for smoother learning.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more