7 Generate high-resolution images with diffusion models

This chapter moves from basic diffusion on small grayscale images to producing vivid, high‑resolution color images efficiently and with greater control. It introduces two pillars that underpin modern text‑to‑image systems: a deeper U‑Net augmented with attention to scale modeling capacity, and the DDIM noise scheduler to accelerate sampling without sacrificing visual quality. It also highlights how the initial noise seed governs image outcomes and shows how interpolating between seeds yields smooth, controllable transitions—skills that prepare you for latent diffusion and Stable Diffusion later in the book.

On the modeling side, the denoising U‑Net is expanded with more downsampling and upsampling blocks, richer skip connections, and SDPA attention placed in both encoder and decoder to prioritize salient spatial features amid growing depth and redundancy. Time information is injected via sinusoidal embeddings, and forward diffusion is performed with DDIM to create training pairs of clean and noisy images. The model is trained on the Oxford 102 Flowers dataset by predicting the added noise at randomly chosen timesteps, optimized with AdamW and a cosine learning‑rate schedule, enabling it to reconstruct clean images from noise while retaining high‑frequency detail.

For inference, DDIM’s deterministic, non‑Markovian schedule enables high‑quality samples in far fewer steps than DDPM, making generation practical at higher resolutions. Beyond single images, the chapter demonstrates seed control and noise interpolation: blending two initial noise tensors produces sequences that morph smoothly between subjects, supporting controlled image synthesis and creative effects like transitions and animations. Together, attention‑enhanced U‑Nets, efficient DDIM sampling, and noise interpolation substantially elevate quality, speed, and controllability, laying the groundwork for text‑conditioned models explored next.

Eight steps for building a text-to-image generator from scratch. In this chapter, you will focus on mastering steps 3 and 4: enhancing the U-Net architecture with attention and leveraging advanced noise scheduling with DDIM. By incorporating attention into a deeper U-Net, you empower your diffusion models to generate highly detailed images that can accurately match text prompts in future chapters.

The architecture of the denoising U-Net model to generate high-resolution flower images. The U-Net is characterized by its symmetric U-shape, with the contracting path on the left, and the expansive path on the right, connected by the bottleneck block at the bottom. The model takes noisy images as inputs and predicts the noise in the images.

How image interpolation works in diffusion models. Initial noise tensor A (top left) leads to final output A (top right), while Initial noise tensor B (bottom left) leads to final output B (bottom right). We create an interpolated noise tensor (middle left) by averaging initial noise tensors A and B. We then feed this interpolated noise tensor into the trained denoising U-Net model to obtain an interpolated output image (middle right), which shows the characteristics of both final images A and B.

Sample images from the Oxford 102 Flower dataset, which contains roughly 8000 flower images. We use this dataset to train the denoising U-Net model.

The forward diffusion process using the DDIM noise scheduler. The four images in the leftmost column are clean images from the training dataset. We gradually add noise to the images. The second column shows the images after 200 time steps. The third column shows the noisy images after 400 time steps. The last column shows the images after 1000 time steps, when images are converted to pure random noise.

Ten flower images generated by the trained denoising U-Net model. We feed ten random noise tensors to the trained U-Net and use 50 inference steps to gradually remove noise from the noisy images. The ten images in this figure look all different due to the different initial noise tensors used.

Interpolated flower images. With the random state set to 100, we select the 2nd and 8th initial noise tensors to generate ten interpolated noise tensors by applying varying weightings to the originals. These interpolated tensors are then passed through the trained denoising U-Net, producing a series of ten flower images that smoothly transition from one to the other.

Three sets of interpolated flower images. Using a fixed random state of 100, we generate three pairs of initial noise tensors. Interpolated noise tensors are then created by blending the original tensors with varying weightings. Interpolated noise tensors are fed to the trained U-Net model to generate interpolated images. In each row, the images gradually transition from the leftmost to the rightmost image, illustrating smooth interpolation.

• The denoising diffusion probabilistic models (DDPM) we studied in previous chapters use a randomized process and follow a Markovian process, meaning each step depends only on the immediate previous step. In contrast, a denoising diffusion implicit model (DDIM) takes a more direct and predictable approach. It uses a deterministic, non-Markovian process, allowing it to generate images in fewer steps without sacrificing quality.
• While a simple U-Net may be adequate for generating grayscale images due to the reduced complexity, generating high-definition color images requires a more sophisticated architecture to handle the increased dimensionality and capture complex spatial and color relationships.
• The advanced U-Net architecture in this chapter improves the model's ability to reconstruct fine details during denoising. However, adding more contracting and expansive blocks can lead to redundant feature extractions, making it challenging to identify the most relevant features. To mitigate this issue, the scaled dot product attention (SDPA) mechanism is incorporated, allowing the model to focus on significant features while filtering out the less important ones.
• Once the denoising model is trained, we can generate high-resolution images by inputting an initial noise tensor into the U-Net. This noise tensor serves as a starting point that the model gradually refines into a clear and detailed image. Because the final image is influenced by the specific noise tensor used, different initial noise tensors will yield different results.
• If we start with two different initial noise tensors, we can generate an interpolated tensor by taking a weighted average of them. Feeding this interpolated tensor into the trained denoising U-Net produces an image that blends the characteristics of the two images generated from the original noise tensors.
• By varying the weights assigned to the two original noise tensors, we can create a series of interpolated initial noise tensors. Feeding these tensors into the trained denoising U-Net model generates a sequence of interpolated final images, displaying a smooth transition between the two original images.

FAQ