table of content

Part 1 Understanding attention and transformers

1 A tale of two models: Transformers and diffusions

1.1 What is a text-to-image generation model?

1.1.1 Unimodal vs. multimodal models

1.1.2 Practical use cases of text-to-image models

1.2 Transformer-based text-to-image generation

1.2.1 Converting an image into a sequence of integers and then back

1.2.2 Training and using a transformer-based text-to-image model

1.3 Text-to-image generation with diffusion models

1.3.1 Forward and reverse diffusions

1.3.2 Latent diffusion models and Stable Diffusion

1.4 How to build text-to-image models from scratch

1.5 Challenges for text-to-image generation models

1.5.1 Are generative AI models stealing from artists?

1.5.2 The geometric inconsistency problem

1.6 Social, environmental, and ethical concerns

2 Build a transformer

2.1 An overview of attention and transformers

2.1.1 How the attention mechanism works

2.1.2 How to create a transformer

2.2 Word embedding and positional encoding

2.2.1 Word tokenization with the Spacy library

2.2.2 A sequence padding function

2.2.3 Input embedding from word embedding and positional encoding

2.3 Creating an encoder–decoder transformer

2.3.1 Coding the attention mechanism

2.3.2 Defining the Transformer() class

2.3.3 Creating a language translator

2.4 Training and using the German-to-English translator

2.4.1 Training the encoder–decoder transformer

2.4.2 Translating German to English with the trained model

3 Classify images with a vision transformer

3.1 The blueprint to train a ViT

3.1.1 Converting images to sequences

3.1.2 Training a ViT for classification

3.2 The CIFAR-10 dataset

3.2.1 Downloading and visualizing CIFAR-10 images

3.2.2 Preparing datasets for training and testing

3.3 Building a ViT from scratch

3.3.1 Dividing images into patches

3.3.2 Modeling the positions of different patches in an image

3.3.3 Using the multi-head self-attention mechanism

3.3.4 Building an encoder-only transformer

3.3.5 Using the ViT to create a classifier

3.4 Training and using the ViT to classify images

3.4.1 Choosing the optimizer and the loss function

3.4.2 Training the ViT for image classification

3.4.3 Classifying images using the trained ViT

4 Add captions to images

4.1 Training and using a transformer to add captions

4.1.1 Preparing data and the causal attention mask

4.1.2 Creating and training a transformer

4.2 Preparing the training dataset

4.2.1 Downloading and visualizing Flickr 8k images

4.2.2 Building a vocabulary of tokens

4.2.3 Preparing the training dataset

4.3 Creating a multimodal transformer to add captions

4.3.1 Defining a ViT as the image encoder

4.3.2 Creating the decoder to generate text

4.4 Training and using the image-to-text transformer

4.4.1 Training the encoder–decoder transformer

4.4.2 Adding captions to images with the trained model

Part 2 Introduction to diffusion models

5 Generate images with diffusion models

5.1 The forward diffusion process

5.1.1 How diffusion models work

5.1.2 Visualizing the forward diffusion process

5.1.3 Different diffusion schedules

5.2 The reverse diffusion process

5.3 A blueprint to train the U-Net model

5.3.1 Steps in training a denoising U-Net model

5.3.2 Preprocessing the training data

5.4 Training and using the diffusion model

5.4.1 The Denoising Diffusion Probabilistic Model noise scheduler

5.4.2 Inference using the U-Net denoising model

5.4.3 Training and using the denoising U-Net model

6 Control what images to generate in diffusion models

6.1 Classifier-free guidance in diffusion models

6.1.1 An overview of classifier-free guidance

6.1.2 A blueprint to implement CFG

6.2 Different components of a denoising U-Net model

6.2.1 Time step embedding and label embedding

6.2.2 The U-Net denoising model architecture

6.2.3 Down blocks and up blocks in the U-Net

6.3 Building and training the denoising U-Net model

6.3.1 Building the denoising U-Net

6.3.2 The Denoising Diffusion Probabilistic Model

6.3.3 Training the diffusion model

6.4 Generating images with the trained diffusion model

6.4.1 Visualizing generated images

6.4.2 How the guidance parameter affects generated images

7 Generate high-resolution images with diffusion models

7.1 Attention in U-Net, DDIM, and image interpolation

7.1.1 Incorporating the attention mechanism in the U-Net model

7.1.2 Denoising Diffusion Implicit Models

7.1.3 Image interpolation in diffusion models

7.2 High-resolution flower images as training data

7.2.1 Visualizing images in the training dataset

7.2.2 Applying forward diffusion on flower images

7.3 Building and training a U-Net for high-resolution images

7.3.1 Building the denoising U-Net model

7.3.2 Training the denoising U-Net model

7.4 Image generation and interpolation

7.4.1 Using the trained denoising U-Net to generate images

7.4.2 Transition from one image to another

Part 3 Text-to-image generation with diffusion models

8 CLIP: A model to measure the similarity between image and text

8.1 The CLIP model

8.1.1 How the CLIP model works

8.1.2 Selecting an image from Flickr 8k based on a text description

8.2 Preparing the training dataset

8.2.1 Image-caption pairs in Flickr 8k

8.2.2 The DistilBERT tokenizer

8.2.3 Preprocess captions and images for training

8.3 Creating a CLIP model

8.3.1 Creating a text encoder

8.3.2 Creating an image encoder

8.3.3 Building a CLIP model

8.4 Training and using the CLIP model

8.4.1 Training the CLIP model

8.4.2 Using the trained CLIP model to select images

8.4.3 Using the OpenAI pretrained CLIP model to select images

9 Text-to-image generation with latent diffusion

9.1 What is a latent diffusion model?

9.1.1 How variational autoencoders work

9.1.2 Combining a latent diffusion model with a variational autoencoder

9.2 Compressing and reconstructing images with VAEs

9.2.1 Downloading the pretrained VAE

9.2.2 Encoding and decoding images with the pretrained VAE

9.3 Text-to-image generation with latent diffusion

9.3.1 Guidance by the CLIP model

9.3.2 Diffusion in the latent space

9.3.3 Converting latent images to high-resolution ones

9.4 Modifying existing images with text prompts

10 A deep dive into Stable Diffusion

10.1 Generating images with Stable Diffusion

10.2 The Stable Diffusion architecture

10.2.1 Generating images from text with Stable Diffusion

10.2.2 Text embedding interpolation

10.3 Creating text embeddings

10.4 Image generation in the latent space

10.5 Converting latent images to high-resolution ones

Part 4 Text-to-image generation with transformers

11 VQGAN: Convert images into sequences of integers

11.1 Converting images into sequences of integers and back

11.2 Variational autoencoders

11.2.1 What is an autoencoder?

11.2.2 The need for VAEs and their training methodology

11.3 Vector quantized variational autoencoders

11.3.1 The need for VQ-VAEs

11.3.2 The VQ-VAE model architecture and training process

11.4 Vector quantized generative adversarial networks

11.4.1 Generative adversarial networks

11.4.2 VQGAN: A GAN with a VQ-VAE generator

11.5 A pretrained VQGAN model

11.5.1 Reconstructing images with the pretrained VQGAN

11.5.2 Converting images into sequences of integers

12 A minimal implementation of DALL-E

12.1 How min-DALL-E works

12.1.1 Training min-DALL-E

12.1.2 From prompt to pixels: Image generation at inference time

12.2 Tokenizing and encoding the text prompt

12.2.1 Tokenizing the text prompt

12.2.2 Encoding the text prompt

12.3 Iterative prediction of image tokens

12.3.1 Loading the pretrained BART decoder

12.3.2 Predicting image tokens using the BART decoder

12.4 Converting image tokens to high-resolution images

12.4.1 Loading the pretrained VQGAN detokenizer

12.4.2 Visualizing the intermediate and final high-resolution outputs

Part 5 New developments and challenges

13 New developments and challenges in text-to-image generation

13.1 State-of-the-art text-to-image generators

13.1.1 DALL-E series

13.1.2 Google’s Imagen

13.1.3 Latent diffusion models: Stable Diffusion and Midjourney

13.2 Challenges and concerns

13.3 A blueprint to fine-tune ResNet50

13.3.1 The history and architecture of ResNet50

13.3.2 A plan to fine-tune ResNet50 for classification

13.3.3 Using ResNet50 to classify images

13.4 Fine-tuning ResNet50 to detect fake images

13.4.1 Downloading and preprocessing real and fake face images

13.4.2 Fine-tuning ResNet50

13.4.3 Detecting deepfakes using the fine-tuned ResNet50

Appendix

Installing PyTorch and enabling GPU training locally and in Colab

A.1 Installing Python and setting up a virtual environment

A.1.1 Installing Anaconda

A.1.2 Setting up a Python virtual environment

A.1.3 Installing Jupyter Notebook

A.2 Installing PyTorch

A.3 Using Google Colab for GPU training and inference

references

Overview

10 A deep dive into Stable Diffusion

This chapter presents a high-level, practical tour of Stable Diffusion, explaining how it refines latent diffusion to deliver fast, flexible, and high‑quality text‑to‑image generation. Built by CompVis, Stability AI, and LAION, the model scales the original latent diffusion approach with optimizations and a curated subset of the larger LAION‑5B dataset, enabling richer visual concepts while remaining openly available. A central theme is accessibility: through Hugging Face’s diffusers library, Stable Diffusion can be used in just a few lines of code, with simple controls for image size and classifier‑free guidance, making state‑of‑the‑art image synthesis approachable for both newcomers and practitioners.

Under the hood, the system combines three components: a CLIP text encoder that turns prompts into fixed‑length token embeddings, a conditional denoising U‑Net that iteratively removes noise in a compact latent space, and a VAE decoder that reconstructs final high‑resolution images. Operating in the latent domain dramatically reduces memory and compute cost while preserving detail through the VAE. The chapter walks through the end‑to‑end pipeline: tokenization to 77 tokens, 768‑dimensional embeddings, reverse diffusion over a set number of steps (e.g., 50), and decoding to RGB images. It also clarifies guidance via unconditional/conditional mixing (CFG) and the role of schedulers like DDIM and LMS, as well as practical details such as latent scaling (the 0.18215 factor) to keep training and inference distributions aligned.

From a user’s perspective, the chapter demonstrates image generation with StableDiffusionPipeline, batch prompting, resolution and guidance tuning, and GPU memory considerations. It then drills into the components directly: creating CLIP embeddings (including unconditional ones), generating latents with the K‑LMS scheduler, and decoding with the VAE, along with simple visualization utilities. A highlight is text‑embedding interpolation: blending embeddings from two prompts (e.g., “a lion” and “a tiger”) to produce smooth semantic transitions in the outputs. This technique offers creative control for hybrid imagery, parameter‑swept variations, and even animations, rounding out a clear, hands‑on deep dive into how Stable Diffusion turns language into images.

Eight steps for building a text-to-image generator from scratch. Here, we focus on step 6, Stable Diffusion, showing how this state-of-the-art generator builds directly on the latent diffusion models introduced in chapter 9. You’ll learn not just how to use Stable Diffusion, but also how to manipulate the text embeddings to create novel images.

An image generated by the StableDiffusionPipeline package of the diffusers library using the text prompt "an astronaut in a spacesuit riding a unicorn."

Three images generated by Stable Diffusion. The three text prompts are "a panda eating a bowl of noodles", "a dog with sunglasses and a straw hat", and "a cat stretching on the floor", respectively.

An image generated by Stable Diffusion using the prompt "dogs running on a sandy beach under blue sky." The image resolution is 768x512 and the classifier-free guidance parameter is set to 12.

How Stable Diffusion converts text prompts to images. Stable Diffusion has three main components: a text encoder, a U-Net, and a VAE decoder. The text encoder converts text prompts such as “a lion” or “a tiger” into text embeddings. The U-Net conducts the reverse diffusion process in the latent space, using the text embeddings as the conditional information. The generated latent images are then converted by the VAE decoder into high-resolution images as outputs.

Using text embedding interpolation to create hybrid images in Stable Diffusion. Suppose you have two different prompts, “a lion” and “a tiger.” We first use the CLIP text encoder to convert them into two text embeddings, A, and B. We can blend A and B and create an interpolated text embedding, which is then used to guide the U-Net in the reverse diffusion process in the latent space. The generated latent images are then converted into high-resolution images as the final output, exhibiting characteristics of both a lion and a tiger (bottom right).

Latent images and final outputs from Stable Diffusion. We use “a lion” and “a tiger” as the text prompts. The Stable Diffusion model first converts the two prompts into two text embeddings. The model then conducts the reverse diffusion process in a latent space, guided by the text embeddings. The generated latent images are shown as the first and the third image in this figure. The latent images are converted to high-resolution ones by using a VAE decoder. The high-resolution final outputs are shown as the second and the fourth images in this figure.

The text embeddings for “a lion” and “a tiger” are combined with different weights to form a series of interpolated text embeddings. The Stable Diffusion model then conducts the reverse diffusion process in the latent space, guided by the interpolated text embeddings. The latent images are converted to high-resolution ones by using a VAE decoder. The high-resolution final outputs are shown in this figure, with different weights placed on the two text embeddings.

Summary

Stable Diffusion is based on the latent diffusion model (LDM) that we discussed in the previous chapter. It's one of the state-of-the-art text-to-image models, and the only one that is open-source and free for anyone to use.
Stable Diffusion is developed jointly by CompVis, Stability AI, and LAION. Stable Diffusion has incorporated improvements that enhance both the model’s stability and performance, such as advanced training techniques and optimizations. While the original LDM was trained on the LAION-400M dataset (with 400 million image-text pairs), Stable Diffusion was trained on a subset of the even larger LAION-5B database.
The StableDiffusionPipeline package in the diffusers library allows us to use Stable Diffusion as an off-the-shelf tool to generate extraordinary images in just a few lines of code.
By blending the embeddings of two different text prompts, we can create an interpolated text embedding. If we use this embedding to guide the latent diffusion process, we can obtain an interpolated latent image, which can in turn be converted to an interpolated final output image. The output image will have characteristics of the two original text prompts. For example, interpolating between the embeddings for “a lion” and “a tiger” will produce an image with characteristics of both a lion and a tiger.

FAQ

How does Stable Diffusion differ from the original Latent Diffusion Model (LDM)?

Stable Diffusion builds directly on the LDM framework but adds extensive optimizations and uses a curated subset of the larger LAION-5B dataset (vs. LAION-400M for the original LDM). It emphasizes accessibility: open-source weights, easy integration with libraries like diffusers, faster inference, and higher-quality, more diverse generations.

How is Stable Diffusion integrated into Hugging Face diffusers?

The StableDiffusionPipeline in diffusers wraps all core steps—tokenization, text encoding, latent denoising, and VAE decoding—so you can generate images in just a few lines. You load a model via from_pretrained (e.g., “CompVis/stable-diffusion-v1-4”), optionally use fp16 for speed, set parameters like width, height, and guidance_scale, and pass a prompt or a list of prompts. It runs efficiently on a CUDA GPU and can be tried in Google Colab.

What are the main components of Stable Diffusion and their roles?

Stable Diffusion has three core parts: (1) CLIP text encoder: converts your prompt into a fixed-length embedding; (2) conditional denoising U-Net: runs the reverse diffusion process in latent space, guided by the text embedding; (3) VAE decoder: converts the final latent into a high-resolution RGB image.

How does the CLIP text encoder handle prompts?

Prompts are tokenized up to a fixed length of 77 tokens; shorter prompts are padded, longer ones are truncated. Each token becomes a 768-dimensional vector, producing a (batch, 77, 768) embedding tensor. These embeddings condition the U-Net during denoising.

What happens during image generation in the latent space?

Generation starts from random noise with latent shape (4, 64, 64) for a 512×512 image. Over a sequence of inference steps (e.g., 50), the U-Net removes noise conditioned on the text embedding, progressively shaping the latent into a coherent representation. The VAE decoder then turns this latent into a 3×512×512 image.

Why is there a 0.18215 scaling factor for latents?

During training, latents are scaled by 0.18215 so their standard deviation is near 1, stabilizing training. At inference, dividing by 0.18215 reverses this normalization before VAE decoding, keeping the distribution consistent with what the VAE learned and improving reconstruction quality.

What is classifier-free guidance (CFG), and how do I choose guidance_scale?

CFG mixes unconditional and conditional predictions to control adherence to the prompt. guidance_scale tunes this balance: higher values (e.g., 12) follow the prompt more strictly but can reduce diversity; lower values introduce more creativity but may drift from the prompt. The diffusers default is 7.5, a good starting point.

Which scheduler should I use (LMS vs. DDIM), and what changes?

Schedulers define how noise is removed across steps. DDIM can be faster with fewer steps but may slightly reduce fidelity. LMS (K-LMS in diffusers) often yields higher visual quality, especially for complex or higher-resolution images, at some compute cost. Choose DDIM for speed, LMS for quality.

How do resolution and batching affect memory and speed?

Higher resolutions and generating multiple images at once significantly increase GPU memory usage. If you hit out-of-memory errors, lower width/height or generate one image at a time. Using fp16 (variant="fp16", torch_dtype=float16) reduces memory and speeds up inference, with minimal precision loss on modern GPUs.

What is text embedding interpolation, and what can I use it for?

Interpolation blends two text embeddings (e.g., “a lion” and “a tiger”) with a weight α to create intermediate embeddings. Conditioning the U-Net on these produces images that smoothly transition between concepts. It’s useful for fine control over hybrid styles, creative exploration, and even animations that morph between ideas.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more