4 Understanding How Stable Diffusion Works

The chapter begins by clarifying the core terms behind modern AI systems and placing them in a hierarchy: artificial intelligence is the broad idea, machine learning is a data-driven approach within it, and neural networks are a specific kind of machine learning. It explains that Stable Diffusion is not a mysterious single invention, but a collection of interconnected components, much like a computer with separate parts. The key idea is that these systems learn from examples, and that understanding their structure will matter more and more as the book moves toward customization.

From there, the text gives a high-level picture of how Stable Diffusion works. In training, the model is shown images with captions and learns to predict the noise that has been added to them, rather than directly predicting the image itself. In generation, the process is reversed: the model starts from pure noise and repeatedly removes noise step by step, guided by the text prompt, until an image emerges. This is the central trick of diffusion models, and the chapter compares it to seeing shapes in clouds, where a vague noisy pattern gradually becomes a recognizable scene.

The rest of the chapter breaks down the major pieces that make this possible. A variational autoencoder compresses images into latent space so the model can work efficiently while still generalizing beyond exact copies; a CLIP-based text system tokenizes prompts and converts them into a dense numerical form that helps the model connect language to visual meaning; a U-NET estimates the noise in the latents; and a scheduler controls how denoising happens over multiple steps. The chapter closes by showing these components in a Stable Diffusion pipeline and emphasizing that because the system is open source, each part can be swapped or tuned to improve results for different creative tasks.

Visualizing the hierarchy of AI related terms

AI systems are built from components much the same way a PC is.

A simple neural network that computes if both A and B are on.

In generate mode Stable Diffusion takes text and turns it into an image.

In learn mode Stable Diffusion works like a bad photocopier

Classifiers try to predict what the object in an image is.

It’s not too hard to figure what in this image is added noise

Stable Diffusion tries to guess that noise in added to an image

Guessing what the original image was by removing noise

Map of the inner workings of Stable Diffusion

Closer look at the variational autoencoder.

Imperfect memory can be a feature, not a bug, when creating images

Image from the film Metropolis

The VAE transforms the image into latents

Reconstructed image, close but the eyes are a bit off.

Comparing noise in the raw image to noise in the latents.

Closer look at the CLIP encoding.

Tokenizing by assigning a number to each character (including spaces).

Tokenizing by assigning a number to each word.

The tokenizer learning the best way to split up the text.

The U-NET is so named because it looks like a “U”

• AI systems are able to learn to improve themselves at solving a task, in this case image generation.
• Machine learning requires us to represent our problem as numbers and math equations.
• Stable Diffusion works by learning to denoise images, then, when generating, is given noise and asked to remove noise from it to create an image.
• The VAE is able to transform images into compressed “latents” that not only make Stable Diffusion more efficient, but help it generalize.
• Most of the time Stable Diffusion is working with latents, not the images themselves.
• Text is broken down into “tokens” which can then be represented numerically. Stable Diffusion learns the best way to break down text.
• A scheduler controls how Stable Diffusion removes noise in order to create an image.

FAQ