9 Text-to-image generation with latent diffusion

This chapter introduces latent diffusion models (LDMs) as a practical and efficient way to generate high-quality images from text. Instead of operating directly in pixel space—which is computationally expensive at high resolution—LDMs perform forward and reverse diffusion in a compact latent space learned by a variational autoencoder (VAE). A high-resolution image (for example, 3×512×512) is compressed to a much smaller representation (such as 4×64×64), cutting computation dramatically while preserving semantic content. After denoising in this latent space, a VAE decoder reconstructs detailed, high-resolution outputs. Beyond text-to-image synthesis, the approach also supports image editing guided by text prompts while maintaining scene structure.

The LDM architecture combines three components: a VAE for compression and reconstruction, a U-Net diffusion model for denoising in the latent space, and a CLIP model for text conditioning. Training proceeds on image–caption pairs: images are encoded into latents, noise is progressively added, and the diffusion model learns to predict and remove that noise conditioned on text. CLIP supplies text and image embeddings to align generated latents with the prompt. This division of labor balances efficiency and fidelity—the VAE shapes a smooth, informative latent manifold and restores fine details, while diffusion ensures realism and diversity. Because end-to-end training is resource-intensive, the chapter focuses on using pre-trained components to demonstrate the full pipeline.

Practically, the chapter walks through compressing and reconstructing images with a pre-trained VAE, then generating images by sampling in latent space with a noise scheduler and time embeddings. Text prompts are tokenized to a fixed length for CLIP, and classifier-free guidance blends conditional and unconditional predictions, with a cfg_scale parameter trading off diversity (lower values) against prompt adherence (higher values). For image-to-image editing, the original image is encoded to latents, a controllable amount of noise (strength) determines how much it changes, and guided denoising steers the result toward the new text description. Decoding intermediate and final latents back to pixel space reveals the gradual emergence of structure and detail, culminating in high-resolution images that accurately reflect the prompts.

Eight steps for building a text-to-image generator from scratch. This chapter lays the groundwork for step 6, Latent Diffusion, showing you how to generate images in a lower-dimensional latent space, guided by text using the CLIP model from the previous chapter.

How a variational autoencoder (VAE) compresses images to representations in the lower-dimensional (latent) space and then reconstructs the images. A VAE consists of an encoder and a decoder. The encoder compresses images (e.g., with a size of 3x512x512) into a lower-dimensional space (i.e., the latent space, where images have a size of, e.g., 4x64x64). The decoder takes the encoded images in the latent space and reconstructs the images to the original high resolution space (with a size of 3x512x512 in this example).

How a trained latent diffusion model (LDM) generates an image based on a text prompt. The text prompt ("as shown at the top left corner) is first encoded into a text embedding. To generate an image in the lower-dimensional space (latent space), we start with an image of pure noise (bottom left). We use the trained U-Net to iteratively denoise the image, conditional on the text embedding so the generated image matches the text prompt, with the guidance of a trained contrast language-image pre-training (CLIP) model. The generated latent image (bottom right) is presented to a trained VAE to convert it to a high-resolution image, which is the final output (top right).

Modifying an existing image with a text prompt using the pre-trained latent diffusion model. The original image is shown in the middle of the figure. Using "a white horse walks on the sunny sandy beach" as the prompt, the resulting image is shown on the left. When we use "a zebra walks on the white, snow-covered ground" as the text prompt, the resulting image is shown on the right.

Comparing the original images with latent images. The 1st and 3rd images in the figure are original images of a beaver and a dolphin, respectively, with a shape of 3x512x512. We use a pre-trained VAE encoder to compress the images to a lower-dimensional space. The compressed latent images have a shape of 4x64x64, with only about 2% of the number of pixels of the original images. However, we can still tell that the 2nd image is a beaver and the last image is a dolphin.

Comparing the original images with reconstructed images. The 1st and 3rd images in the figure are original images of a beaver and a dolphin, respectively, with a shape of 3x512x512. We use a pre-trained VAE encoder to compress the images into lower-dimensional latent images with a shape of 4x64x64. We then use the pre-trained VAE decoder to convert the latent images to the original high-dimensional space, as shown in the 2nd and 4th images in the figure.

An image generated by a pre-trained latent diffusion model with a text prompt " a banana riding a motorcycle, wearing sunglasses and a straw hat, highly detailed, ultra sharp, cinematic, 100mm lens, 8k resolution."

Generated latent images in time steps 800, 600, …, and 0. These images are generated with a pre-trained latent diffusion model. We start with an image with pure random noise at t=1000. We use the diffusion model to denoise it iteratively and obtain the noisy image at t=800, which is shown in the far left of the figure. We continue to denoise the image and obtain images at t=600, 400, and 200. Finally, we obtain the image at t=0, which is shown at the far right of the figure. The images have a shape of 4x64x64.

Converting the intermediate latent images into the high-dimensional space. The five latent images in figure 9.8, each with a shape of 4x64x64, are converted by the pre-trained VAE decoder into the high-dimensional space, with a shape of 3x512x512.

An image of a horse. We’ll use a pretrained latent diffusion model (LDM) to modify this image based on text prompts.

A modified horse image with the latent diffusion model (LDM). We use the text prompt “a white horse walks on the sandy sunny beach” to modify the horse image shown in figure 9.10 using a pretrained LDM.

A modified image with a pretrained latent diffusion model (LDM) based on the text prompt “a zebra walks on the white, snow-covered ground.” The original image is a white horse as shown in figure 9.10.

An image generated by a trained latent diffusion model based on a text prompt when we set the cfg_scale parameter to 12. The text prompt is " a banana riding a motorcycle, wearing sunglasses and a straw hat, highly detailed, ultra sharp, cinematic, 100mm lens, 8k resolution." The classifier-free guidance parameter, cfg_scale, steers the generation process towards outputs that better match the provided conditional prompt, while reducing artifacts. A low value of the cfg_scale parameter (say, 5) leads to a more diverse output, and a high value (say, 12) leads to an output that aligns more closely with the text prompt.

• Generating high-resolution images with diffusion models conditional on text prompts is computationally costly, making training and inference inefficient and slow. To overcome the challenge, Rombach et al. in 2022 proposed a latent diffusion model (LDM). Instead of conducting forward and reverse diffusion processes on high-resolution images, the authors propose conducting the diffusion processes in a lower-dimensional space (i.e., the latent space).
• Since images in the latent space are much smaller in size, the diffusion processes are much faster. However, this means lower quality of the generated images. To compensate for this, a variational autoencoder (VAE) is used to convert the low-resolution latent images back into high-resolution ones as the final output.
• A VAE is a generative model composed of an encoder and a decoder. The encoder transforms high-resolution images into compact, lower-dimensional representations, which the decoder uses to reconstruct the original image. The decoder’s ability to convert low-resolution images into high-resolution output is an essential part of the LDM.
• An LDM can also modify an existing image based on a text prompt, similar to how it generates an image from scratch, but with a key difference: you start from an encoded version of the image rather than pure noise.
• The classifier-free guidance parameter, cfg_scale, controls the diversity versus fidelity of generated images. A low cfg_scale value leads to more diverse output. A high cfg_scale value leads to an output that is more aligned with the text prompt.
• One of the state-of-the-art text-to-image generators, Stable Diffusion, which we’ll cover in the next chapter, is based on LDM.

FAQ