Overview
15 Diffusion Models and Text-to-Image Transformers
This chapter introduces diffusion models as the backbone of modern text-to-image systems within the broader rise of multimodal LLMs. It explains forward diffusion—gradually corrupting clean images with Gaussian noise across many discrete time steps—and reverse diffusion—learning to remove that noise to synthesize new samples—grounded by a hands-on example that generates flower images. The aim is to build intuition for why diffusion delivers high-resolution, diverse, and faithful outputs.
You implement the full pipeline in PyTorch: a denoising U-Net with a symmetric encoder–decoder, skip connections, and scaled dot-product attention, all conditioned on sinusoidal time-step embeddings. Training pairs clean images with synthetically noised counterparts, and the model learns to predict the injected noise using an L1 loss while AdamW and a cosine learning-rate schedule guide optimization. A DDIM-style scheduler controls both noise addition and sampling; after training, the model iteratively denoises random noise over a fixed number of inference steps to produce realistic 64×64 flower images.
Building on this foundation, the chapter explains how text-to-image Transformers condition diffusion on language. It details CLIP’s dual encoders that align text and images in a shared embedding space via contrastive learning, and shows how systems like DALL-E 2 use text embeddings and a CLIP prior to guide the U-Net during iterative denoising. A short program demonstrates prompting an image generator via an API, tying the conceptual pieces to practical text-to-image creation.
A diagram of the forward diffusion process. We start with a clean image from the training set, 𝑥0, and add noise ϵ0 to it to form a noisy image 𝑥1 = √(1- 𝛽1)𝑥0 + √(𝛽1)ϵ0. We repeat this process for 1000 time steps until the image 𝑥1000 becomes random noise.
The architecture of the denoising U-Net model. The U-Net architecture is characterized by its symmetric shape, with a contracting path (encoder) and an expansive path (decoder), connected by a bottleneck layer. The model is designed to remove noise from images while preserving important details. The input to the model is a noisy image, along with which time step the image is in, and the output is the predicted noise in the image.
The training process of the denoising U-Net model. We first obtain clean flower images as our training set. We add noise to clean flower images and present them to the U-Net model. The model predicts the noise in the noisy images. We compare the predicted noise with the actual noise injected into the flower images and tweak the model weights to minimize the mean absolute error.
The first 16 Images from the Oxford 102 Flower dataset.
The forward diffusion process. The four images in the first column are clean images from the training dataset. We then gradually add noise to these images from time step 1 to time step 1000. As the time step increases, more and more noise is injected into the images. The four images in the second column are images after 200 time steps. The third column contains images after 400 time steps and they have more noise than those in the second column. The last column contains images after 1000 time steps and they are 100% random noise.
Flower images created by the trained denoising U-Net model.
How the trained denoising U-Net model gradually converts random noise into clean flower images. We feed random noise to the trained model to obtain the image at time step 980. We then feed the noisy image at t=980 to the model to obtain the image at t=960. We repeat this process 50 inference steps until we obtain the image at t=0. The first column in this figure shows the four flowers at t=800; the second column shows the same four flowers at t=600…; the last column shows the four flowers at t=0 (i.e., clean flower images).
How OpenAI’s CLIP (Contrastive Language-Image Pre-training) model is trained. A large-scale training dataset of text-image pairs is collected. The text encoder of the model compresses the text description into a D-value text embedding. The image encoder converts the corresponding image into an image embedding also with D values. During training, a batch of N text-image pairs are converted to N text embeddings and N image embeddings. CLIP uses a contrastive learning approach to maximize the similarity between paired embeddings (the sum of diagonal values in the figure) while minimizing the similarity between embeddings from non-matching text-image pairs (the sum of off-diagonal values in the figure).
How text-to-image Transformers such as DALL-E 2 create images based on text prompts. The text encoder in the trained text-to-image Transformer first converts the text description in the prompt into text embedding. The text embedding is fed to the CLIP model to obtain a prior vector which represents the image in the latent space. The text embedding and the prior are concatenated into a conditioning vector. To generate an image, the U-Net denoiser first takes a random noise vector as input to generate a noisy image using the conditioning vector. It then takes the noisy image and the conditioning vector as input and generates another image, which is less noisy. The process is repeated for many iterations until the final output, a clean image, is generated.
An image generated by DALL-E 2 with the text prompt "an astronaut in a space suit riding a unicorn."
Summary
- In forward diffusion, we gradually add small amounts of random noise to clean images until they transform into pure noise. Conversely, in reverse diffusion, we begin with random noise and employ a denoising model to progressively eliminate noise from the images, transforming the noise back into a clean image.
- The U-Net architecture, originally designed for biomedical image segmentation, has a symmetric shape with a contracting encoder path and an expansive decoder path, connected by a bottleneck layer. In denoising, U-Nets are adapted to remove noise while preserving details. Skip connections link encoder and decoder feature maps of the same spatial dimensions, helping to preserve spatial information like edges and textures that may be lost during downsampling in the encoding process.
- Incorporating an attention mechanism into a denoising U-Net model enables it to concentrate on important features and disregard irrelevant ones. By treating image pixels as a sequence, the attention mechanism learns pixel dependencies, similar to how it learns token dependencies in natural language processing (NLP). This enhances the model's ability to identify relevant features effectively.
- Text-to-image Transformers like OpenAI's DALL-E 2, Google's Imagen, and Stability AI's Stable Diffusion use diffusion models to create images from textual descriptions. They encode the text into a latent representation that conditions the diffusion model, which then iteratively denoises a random noise vector, guided by the encoded text, to generate lifelike images matching the textual description.
FAQ
What are forward and reverse diffusion in image generation?
Forward diffusion gradually corrupts a clean image by adding small amounts of Gaussian noise over many steps (typically T=1000), turning it into nearly pure noise. Reverse diffusion does the opposite: a trained model starts from random noise and iteratively removes noise step by step to synthesize a clean, realistic image.Why does the model predict noise instead of directly predicting the clean image?
In the diffusion formulation, a noisy image at step t is a weighted mix of the clean image and Gaussian noise. If the model predicts the noise, we can algebraically recover the clean image. Predicting noise yields a simple, stable training objective and works well in practice.How does the denoising U-Net architecture work?
A denoising U-Net has a symmetric encoder-decoder design with skip connections. The encoder downsamples to capture global context; the decoder upsamples to reconstruct details. Skip connections pass fine-grained features from encoder to decoder, helping preserve edges and textures. The network takes a noisy image x_t and the time step t as inputs and outputs the predicted noise.What role does scaled dot-product attention (SDPA) play in the U-Net?
SDPA is inserted in the deeper blocks of both the encoder and decoder. By attending across spatial locations (treating flattened pixels as a sequence), it helps the model focus on the most relevant regions and features, improving denoising quality, especially given the U-Net’s large capacity.How are time steps represented and used during denoising?
Time steps are embedded via sinusoidal functions (similar to Transformer positional encodings) into a vector that’s projected and broadcast to match feature-map shapes. These embeddings are added to features so the model “knows” how much noise is present at step t and can predict the appropriate noise to remove.How do you prepare the data and train the denoising model?
- Data: Oxford 102 Flowers, resized to 64×64 with 3 channels, normalized to [-1, 1], small batch size (e.g., 4).- Training loop: For each image, sample a random t, add noise to create x_t, predict noise with the U-Net, and minimize L1 loss between predicted and true noise.
- Optimization: AdamW optimizer with a cosine learning-rate schedule and warmup; L1 loss is preferred for robustness to outliers.
Learn Generative AI with PyTorch ebook for free
