6 Control what images to generate in diffusion models

This chapter shows how to move from unguided image synthesis to controllable generation with conditional diffusion models. It explains how conditioning lets you request specific outputs (such as a coat or sandal) and introduces classifier-free guidance (CFG), a technique that removes the need for an external classifier by jointly training the model with and without conditioning. By interpolating between conditional and unconditional predictions, CFG provides a simple, effective control knob—the guidance scale—to trade off fidelity to the request against output diversity, laying the groundwork for later text-to-image systems.

To make this concrete, the chapter builds a denoising U-Net from scratch and turns it into a conditional generator by injecting two kinds of embeddings: time-step embeddings (to identify the diffusion step) and label embeddings (to specify the desired class). The U-Net follows the classic contracting–bottleneck–expansive design with skip connections; down blocks use convolutions and max pooling to compress features, while up blocks use transposed convolutions to reconstruct detail. The chapter demystifies these components with hands-on, numerical illustrations of max pooling and transposed convolution, then shows how label masking during training enables the same network to act as both conditional and unconditional model—an essential ingredient for CFG.

A DDPM framework ties everything together: a noise scheduler corrupts clean images (FashionMNIST), the U-Net predicts the noise, and training minimizes mean squared error between true and predicted noise. At sampling time, the procedure runs both conditional and unconditional passes and blends them with a guidance weight to steer results; higher weights tighten adherence to the label at the cost of variety, while lower weights do the opposite. By the end, you can train a conditional diffusion model that generates requested classes on demand and modulate the fidelity–diversity balance, establishing the technical foundation that later chapters extend to text prompts and higher-resolution, color image synthesis used in modern latent diffusion systems.

The eight-step roadmap for building a text-to-image generator from scratch. This chapter zeroes in on step 4, empowering your diffusion model to generate specific types of images on command using classifier-free guidance, a breakthrough technique used in state-of-the-art text-to-image generation models.

A diagram of classifier-free guidance. In classifier-free guidance, we jointly train a conditional (top left part of the figure) and unconditional (bottom left part) diffusion model to attain a trade-off between sample quality and diversity. By embedding the conditioning information directly in the diffusion model, we can use the model as either a conditional model or an unconditional model. At inference, classifier-free guidance is applied by interpolating between the model’s conditional and unconditional predictions. The guidance parameter, w, controls how closely the generated image adheres to the conditioning information.

How to embed the time steps and the class labels in the denoising U-Net model. When training the U-Net, we add noise to clean images in the training set. We then feed the noisy images, along with the time steps the noisy images are in and the class labels, to the U-Net to train it to predict the noise in the images. The embedding blocks in the U-Net convert a time step, such as 25, into two time embeddings, with dimensions (512, 1, 1) and (256, 1, 1), respectively. Similarly, a class label, such as 2 (which means the image is a pullover), is converted into two label embeddings, with dimensions (512, 1, 1) and (256, 1, 1), respectively. The time embeddings and label embeddings will be used in the expansive path (i.e., decoder) of the U-Net.

A diagram illustrating the components of the denoising U-Net model. The U-Net takes noisy images, time steps, and labels as inputs and predicts the noise in the noisy images. The architecture consists of three main parts: a contracting path (encoder), an expansive path (decoder), and a bottleneck connecting the two. The contracting path extracts features and downsamples the images through convolutional and pooling layers, progressively capturing higher levels of abstraction. The expansive path reconstructs the feature maps by upsampling the images and utilizes skip connections to merge corresponding features from the contracting path. Finally, the bottleneck serves as a bridge between the encoder and decoder, processing the most abstracted features before reconstruction.

A numerical example of how max pooling works, with a kernel size of 2x2 and a stride equal to 2.

A numerical example of how transposed convolutional operations work.

Comparing the generated images with real images. The top row shows images generated by the trained denoising U-Net model, with labels 0 to 9, respectively. The bottom row displays real images from the training set, also with labels 0 to 9.

Generated images by the trained denoising U-Net model when the guidance parameter is high (with a value of 5). The first column shows images generated with the label 0 (meaning a t-shirt). The second column is trousers (with the label 1), and so on. The last column shows ankle boots, with labels 9.

• Traditional diffusion models typically rely on external classifiers to guide the image generation process using a technique known as classifier guidance. In 2022, Jonathan Ho and Tim Salimans introduced classifier-free guidance and showed that effective guidance during generation can be achieved using a purely generative model without relying on an external classifier. Their approach involves jointly training a diffusion model under both conditional and unconditional settings.
• The contracting path of a U-Net is designed to learn feature representations and progressively downsample the input image. This path usually consists of several blocks of convolutional layers followed by pooling layers. Convolutional layers extract feature maps by detecting patterns such as edges, textures, or shapes at various levels of abstraction. Pooling layers reduce the spatial dimensions of the feature maps, making the model computationally efficient while capturing more abstract representations.
• The bottleneck path in U-Net serves as a bridge between the contracting and expansive paths. It processes the most abstracted features extracted by the contracting path and prepares them for reconstruction in the expansive path. The bottleneck contains layers with the smallest spatial dimensions and the most condensed information about the image.
• The expansive path of a U-Net upscales the downsampled feature maps and reconstructs the image. This path usually consists of transposed convolutional layers and additional convolutional layers. Transposed convolutional layers upsample the feature maps, effectively filling in gaps and increasing spatial resolution.
• To enhance the reconstruction process in a U-Net, skip connections are used to combine feature maps from the contracting path with those in the expansive path. This ensures that fine-grained details lost during downsampling are retained and incorporated back into the upsampled image.

FAQ