12 Generative Image Models

Generative image models translate natural language into images by learning a “visual grammar” from vast collections of image–caption pairs. Modern systems rely chiefly on diffusion, a counterintuitive process that starts from pure noise and iteratively removes what doesn’t belong until a coherent picture emerges. Trained at scale and often aided by language models, these generators connect words to pixels with remarkable fidelity, and fit within the broader landscape of generative learning alongside GANs, supervised, unsupervised, and reinforcement techniques.

Building such models begins with curating and cleaning diverse, well-captioned datasets while addressing licensing, privacy, and bias. Images are normalized to numeric tensors and paired with text embeddings; forward diffusion creates noisy training examples governed by a noise schedule (betas, alphas) and tracked by timesteps. A U-Net—an encoder–decoder CNN with skip connections—forms the denoising “brain,” fusing image features with timestep and text guidance (in large systems via cross‑attention). The training objective is simple but powerful: predict the exact noise added to an image (MSE loss) and update all weights via backprop. At inference, the process runs in reverse: starting from random noise, the model repeatedly predicts and subtracts noise using the same schedule until an image consistent with the prompt appears.

Quality and control arise from smart choices and add‑ons: dataset diversity and caption richness, timestep count and noise schedules, attention mechanisms, and disciplined training with validation loss or FID and checkpointing. Practical capabilities extend well beyond text‑to‑image: inpainting and outpainting edit or extend scenes; LoRA adapts styles efficiently; high‑resolution fixes and upscalers boost detail; ControlNets and IP‑Adapters enforce structure and style from references; and human‑feedback tuning refines aesthetics. The resulting tools power workflows from concept art, design, and advertising to personalized content and intelligent photo editing, all built on the same scalable diffusion-and-U‑Net foundation.

An analogy showing that Diffusion is about removing to create rather than adding

How Diffusion models fits into machine learning

An example of initial noise

An example of denoising

An example of denoising after more timesteps

An example of denoising after all timesteps

A workflow for training a diffusion model

Preparing image data in the diffusion model training workflow

Cleaning image data for training

Embedding in the diffusion model training lifecycle

Example basic image training data

How images are represented as normalized numbers

How RGB images are represented as matrices

Calculations for adding noise to an image

A progression of adding noise to an image over 1000 timesteps

A view of probability of selection in a normal distribution

Calculations for timestep embeddings

The embeddings for timestep 22

The embeddings for “a happy face”

A view of a feed-forward ANN processing the text embedding

A complete representation of one training data input

Designing the architecture in the Diffusion model training workflow

The difference between an ANN and a CNN

How a CNN filter scans an image

Calculations of a filter on an image for the first pixel

Feature maps produced by a convolutional layer

The U-Net architecture

The U-Net architecture for our example

Denoising in the Diffusion model training workflow

The first pixel of all feature maps produced by the first convolutional layer

Normalized values for the first pixel in all feature maps produced from the first convolutional layer

ReLU vs SiLU activation functions

Calculations for the SiLU activation function

The down-sampling convolutional layer in our U-Net

How filters scan an image with padding

Calculations for the down-sampling convolutional layer in our U-Net

Calculations for producing the final value for the first pixel in the feature map

The bridge convolutional layer in our U-Net

How the text embedding and timestep embedding are injected

Simplistic up-sampling with nearest neighbor interpolation

The up-sampling layer in our U-Net

How skip connection feature maps are combined with the up-sampled feature maps

The final convolutions to produce an noise prediction

Learning in the Diffusion model training lifecycle

Calculations for finding the loss

The ideal loss-over- time for a Diffusion model

Adjusting weights based on loss

The components whose weights are updated with backpropagation

Generating an image in the Diffusion model lifecycle

Starting with pure noise for making a prediction

How a denoised image is predicted

Calculations for finding timestep scale for timestep 1000

Generating a new image by removing noise

The denoised images from timestep 1000 to timestep 750

The denoised images from timestep 1000 to timestep 1

The denoised images when we generate with only 50 timesteps instead of 1000 timesteps

How cross-attention fits into the U-Net architecture

FAQ