table of content

Part 1 Understanding attention and transformers

1 A tale of two models: Transformers and diffusions

1.1 What is a text-to-image generation model?

1.1.1 Unimodal vs. multimodal models

1.1.2 Practical use cases of text-to-image models

1.2 Transformer-based text-to-image generation

1.2.1 Converting an image into a sequence of integers and then back

1.2.2 Training and using a transformer-based text-to-image model

1.3 Text-to-image generation with diffusion models

1.3.1 Forward and reverse diffusions

1.3.2 Latent diffusion models and Stable Diffusion

1.4 How to build text-to-image models from scratch

1.5 Challenges for text-to-image generation models

1.5.1 Are generative AI models stealing from artists?

1.5.2 The geometric inconsistency problem

1.6 Social, environmental, and ethical concerns

2 Build a transformer

2.1 An overview of attention and transformers

2.1.1 How the attention mechanism works

2.1.2 How to create a transformer

2.2 Word embedding and positional encoding

2.2.1 Word tokenization with the Spacy library

2.2.2 A sequence padding function

2.2.3 Input embedding from word embedding and positional encoding

2.3 Creating an encoder–decoder transformer

2.3.1 Coding the attention mechanism

2.3.2 Defining the Transformer() class

2.3.3 Creating a language translator

2.4 Training and using the German-to-English translator

2.4.1 Training the encoder–decoder transformer

2.4.2 Translating German to English with the trained model

3 Classify images with a vision transformer

3.1 The blueprint to train a ViT

3.1.1 Converting images to sequences

3.1.2 Training a ViT for classification

3.2 The CIFAR-10 dataset

3.2.1 Downloading and visualizing CIFAR-10 images

3.2.2 Preparing datasets for training and testing

3.3 Building a ViT from scratch

3.3.1 Dividing images into patches

3.3.2 Modeling the positions of different patches in an image

3.3.3 Using the multi-head self-attention mechanism

3.3.4 Building an encoder-only transformer

3.3.5 Using the ViT to create a classifier

3.4 Training and using the ViT to classify images

3.4.1 Choosing the optimizer and the loss function

3.4.2 Training the ViT for image classification

3.4.3 Classifying images using the trained ViT

4 Add captions to images

4.1 Training and using a transformer to add captions

4.1.1 Preparing data and the causal attention mask

4.1.2 Creating and training a transformer

4.2 Preparing the training dataset

4.2.1 Downloading and visualizing Flickr 8k images

4.2.2 Building a vocabulary of tokens

4.2.3 Preparing the training dataset

4.3 Creating a multimodal transformer to add captions

4.3.1 Defining a ViT as the image encoder

4.3.2 Creating the decoder to generate text

4.4 Training and using the image-to-text transformer

4.4.1 Training the encoder–decoder transformer

4.4.2 Adding captions to images with the trained model

Part 2 Introduction to diffusion models

5 Generate images with diffusion models

5.1 The forward diffusion process

5.1.1 How diffusion models work

5.1.2 Visualizing the forward diffusion process

5.1.3 Different diffusion schedules

5.2 The reverse diffusion process

5.3 A blueprint to train the U-Net model

5.3.1 Steps in training a denoising U-Net model

5.3.2 Preprocessing the training data

5.4 Training and using the diffusion model

5.4.1 The Denoising Diffusion Probabilistic Model noise scheduler

5.4.2 Inference using the U-Net denoising model

5.4.3 Training and using the denoising U-Net model

6 Control what images to generate in diffusion models

6.1 Classifier-free guidance in diffusion models

6.1.1 An overview of classifier-free guidance

6.1.2 A blueprint to implement CFG

6.2 Different components of a denoising U-Net model

6.2.1 Time step embedding and label embedding

6.2.2 The U-Net denoising model architecture

6.2.3 Down blocks and up blocks in the U-Net

6.3 Building and training the denoising U-Net model

6.3.1 Building the denoising U-Net

6.3.2 The Denoising Diffusion Probabilistic Model

6.3.3 Training the diffusion model

6.4 Generating images with the trained diffusion model

6.4.1 Visualizing generated images

6.4.2 How the guidance parameter affects generated images

7 Generate high-resolution images with diffusion models

7.1 Attention in U-Net, DDIM, and image interpolation

7.1.1 Incorporating the attention mechanism in the U-Net model

7.1.2 Denoising Diffusion Implicit Models

7.1.3 Image interpolation in diffusion models

7.2 High-resolution flower images as training data

7.2.1 Visualizing images in the training dataset

7.2.2 Applying forward diffusion on flower images

7.3 Building and training a U-Net for high-resolution images

7.3.1 Building the denoising U-Net model

7.3.2 Training the denoising U-Net model

7.4 Image generation and interpolation

7.4.1 Using the trained denoising U-Net to generate images

7.4.2 Transition from one image to another

Part 3 Text-to-image generation with diffusion models

8 CLIP: A model to measure the similarity between image and text

8.1 The CLIP model

8.1.1 How the CLIP model works

8.1.2 Selecting an image from Flickr 8k based on a text description

8.2 Preparing the training dataset

8.2.1 Image-caption pairs in Flickr 8k

8.2.2 The DistilBERT tokenizer

8.2.3 Preprocess captions and images for training

8.3 Creating a CLIP model

8.3.1 Creating a text encoder

8.3.2 Creating an image encoder

8.3.3 Building a CLIP model

8.4 Training and using the CLIP model

8.4.1 Training the CLIP model

8.4.2 Using the trained CLIP model to select images

8.4.3 Using the OpenAI pretrained CLIP model to select images

9 Text-to-image generation with latent diffusion

9.1 What is a latent diffusion model?

9.1.1 How variational autoencoders work

9.1.2 Combining a latent diffusion model with a variational autoencoder

9.2 Compressing and reconstructing images with VAEs

9.2.1 Downloading the pretrained VAE

9.2.2 Encoding and decoding images with the pretrained VAE

9.3 Text-to-image generation with latent diffusion

9.3.1 Guidance by the CLIP model

9.3.2 Diffusion in the latent space

9.3.3 Converting latent images to high-resolution ones

9.4 Modifying existing images with text prompts

10 A deep dive into Stable Diffusion

10.1 Generating images with Stable Diffusion

10.2 The Stable Diffusion architecture

10.2.1 Generating images from text with Stable Diffusion

10.2.2 Text embedding interpolation

10.3 Creating text embeddings

10.4 Image generation in the latent space

10.5 Converting latent images to high-resolution ones

Part 4 Text-to-image generation with transformers

11 VQGAN: Convert images into sequences of integers

11.1 Converting images into sequences of integers and back

11.2 Variational autoencoders

11.2.1 What is an autoencoder?

11.2.2 The need for VAEs and their training methodology

11.3 Vector quantized variational autoencoders

11.3.1 The need for VQ-VAEs

11.3.2 The VQ-VAE model architecture and training process

11.4 Vector quantized generative adversarial networks

11.4.1 Generative adversarial networks

11.4.2 VQGAN: A GAN with a VQ-VAE generator

11.5 A pretrained VQGAN model

11.5.1 Reconstructing images with the pretrained VQGAN

11.5.2 Converting images into sequences of integers

12 A minimal implementation of DALL-E

12.1 How min-DALL-E works

12.1.1 Training min-DALL-E

12.1.2 From prompt to pixels: Image generation at inference time

12.2 Tokenizing and encoding the text prompt

12.2.1 Tokenizing the text prompt

12.2.2 Encoding the text prompt

12.3 Iterative prediction of image tokens

12.3.1 Loading the pretrained BART decoder

12.3.2 Predicting image tokens using the BART decoder

12.4 Converting image tokens to high-resolution images

12.4.1 Loading the pretrained VQGAN detokenizer

12.4.2 Visualizing the intermediate and final high-resolution outputs

Part 5 New developments and challenges

13 New developments and challenges in text-to-image generation

13.1 State-of-the-art text-to-image generators

13.1.1 DALL-E series

13.1.2 Google’s Imagen

13.1.3 Latent diffusion models: Stable Diffusion and Midjourney

13.2 Challenges and concerns

13.3 A blueprint to fine-tune ResNet50

13.3.1 The history and architecture of ResNet50

13.3.2 A plan to fine-tune ResNet50 for classification

13.3.3 Using ResNet50 to classify images

13.4 Fine-tuning ResNet50 to detect fake images

13.4.1 Downloading and preprocessing real and fake face images

13.4.2 Fine-tuning ResNet50

13.4.3 Detecting deepfakes using the fine-tuned ResNet50

Appendix

Installing PyTorch and enabling GPU training locally and in Colab

A.1 Installing Python and setting up a virtual environment

A.1.1 Installing Anaconda

A.1.2 Setting up a Python virtual environment

A.1.3 Installing Jupyter Notebook

A.2 Installing PyTorch

A.3 Using Google Colab for GPU training and inference

references

Overview

12 A minimal implementation of DALL-E

This chapter introduces a practical, minimal implementation of a DALL-E–style text-to-image generator, showing how transformer-based language modeling and vector-quantized image representations work together to turn natural language into high-resolution images. Building on open-source efforts inspired by OpenAI’s DALL-E—especially DALL-E Mini and its PyTorch port, min-DALL-E—the chapter emphasizes understanding the full pipeline rather than treating it as a black box. Readers follow a concrete prompt through each stage, gaining intuition from intermediate outputs that reveal how the picture gradually emerges token by token.

The method is framed as sequence-to-sequence generation: a pretrained BART encoder converts a prompt into dense text embeddings; a transformer decoder (trained in the DALL-E Mini project) autoregressively predicts a sequence of 256 discrete image tokens; and a pretrained VQGAN decoder reconstructs those tokens into a complete image. Training (as done in DALL-E Mini) uses large-scale image–caption pairs, VQGAN-encoded image tokens as targets, cross-entropy supervision, and a frozen text encoder so the decoder learns to “speak image.” At inference, min-DALL-E tokenizes the prompt, encodes it, generates tokens stepwise from a BOS seed, and detokenizes the result. Working in a compressed, discrete token space makes the approach efficient and conceptually similar to language modeling.

Hands-on sections walk through the essential components: BPE tokenization and padding, initializing and loading pretrained BART encoder/decoder and the VQGAN detokenizer, and configuring sampling with temperature, top-k, and classifier-free guidance via a superconditioning factor. The chapter details the iterative decoding loop over a 16×16 grid of image tokens, attention masking and caching, and capturing intermediate states for visualization. Finally, readers detokenize to pixels, inspect snapshots as the image forms (e.g., after 32, 64, …, 256 tokens), and even animate the full progression—cementing key ideas such as vector quantization, autoregressive decoding, and sampling strategies from prompt to pixels.

Eight steps for building a text-to-image generator from scratch. This chapter focuses on the last step: replicating a transformer-based text-to-image generator such as DALLE.

Training pipeline of the DALL-E Mini model. The model is trained on a large dataset of image-caption pairs to replicate the functionality of OpenAI’s proprietary DALLE. During training, each image is encoded into a sequence of discrete tokens using a pretrained VQGAN encoder. At the same time, the accompanying caption is tokenized and passed through a pretrained BART encoder to produce a sequence of text embeddings. These embeddings, along with the VQGAN-encoded image tokens, are used to train a BART decoder from scratch. The decoder learns to autoregressively predict the image tokens, and the training objective is to minimize the cross-entropy loss between the predicted and actual image token sequences.

The image generation process in min-DALL-E. To generate an image from a text prompt. The prompt is first tokenized and passed through a pretrained BART encoder to produce text embeddings. The trained BART decoder then autoregressively predicts a sequence of 256 image tokens. Generation begins with a special BOS (beginning-of-sequence) token and proceeds one token at a time, appending each predicted token to the input sequence. Once all 256 image tokens are generated, they are passed to a pretrained VQGAN decoder, which reconstructs them into a high-resolution image.

An image generated by the min-DALL-E model with the prompt “panda with top hat reading a book.”

How the min-DALL-E model generates an image based on the prompt "panda with top hat reading a book." The model divides an image into 256 patches, organized in a 16x16 grid. When generating an image based on a text prompt, the model first predicts the top left patch. In the next iteration, the model predicts the patch next to it, based on the first patch and the prompt. The process is repeated until we have all the needed patches in the image. In this figure, the top left subplot shows the output when 32 image patches are generated. The second subplot in the top row shows the output when 64 patches are generated. The rest of the images show the outputs when 96, 128, ..., and 256 patches are generated.

Summary

DALL-E Mini is a project to replicate the proprietary DALL-E model, and min-DALL-E is the PyTorch reimplementation of DALL-E Mini, which was built using the FLAX/JAX deep learning framework.
To convert natural language to images, min-DALL-E first tokenizes and encodes the text prompt using a pretrained BART encoder, which converts the prompt into dense vector embeddings that condition the image generation process.
The min-DALL-E model then autoregressively generates 256 image tokens using a trained BART decoder, where each token corresponds to an image patch in a 16×16 grid.
A pretrained VQGAN decoder then converts the sequence of image tokens into a high-resolution color image by looking up visual features from a learned codebook.
Building the min-DALL-E model from scratch allows us to understand how state-of-the-art text-to-image generators such as DALL-E work. The project provides hands-on experience with encoding, decoding, detokenization, and sampling strategies.

FAQ

What is min-DALL-E, and how is it related to OpenAI’s DALL-E?

min-DALL-E is a compact, open-source PyTorch reimplementation that reuses pretrained weights from the DALL-E Mini project (trained in Flax/JAX). It mirrors the core DALL-E idea: encode a text prompt with a transformer encoder, autoregressively predict a sequence of image tokens with a transformer decoder, and reconstruct pixels with a VQGAN detokenizer. It’s not OpenAI’s proprietary DALL-E, but a practical, lightweight way to study and run a DALL-E–style pipeline.

How does the end-to-end text-to-image pipeline work?

- Tokenize the prompt with a BPE tokenizer (adds special tokens, pads to a fixed length).
- Encode text with a pretrained BART encoder to get dense embeddings.
- Autoregressively predict 256 image tokens (a 16×16 grid) with a BART-style decoder, conditioned on the text embeddings (and previously generated tokens).
- Convert the image tokens to pixels with a pretrained VQGAN detokenizer to obtain the final image.

How is the model trained in DALL-E Mini, and what does this chapter actually run?

Training (DALL-E Mini):
- Use ~15M image–caption pairs.
- Pretrained VQGAN encoder converts each image into 256 discrete tokens.
- Pretrained BART encoder (frozen) embeds the caption; a BART decoder is trained to predict the image tokens autoregressively with cross-entropy loss.
Inference in this chapter (min-DALL-E):
- You don’t train; you load pretrained weights converted from DALL-E Mini and run the full generation pipeline in PyTorch.

What are “image tokens,” and why use a VQGAN codebook?

Image tokens are discrete indices into a learned VQGAN codebook. Each token corresponds to a small image patch. Using tokens (rather than raw pixels) makes the problem more tractable for transformers and aligns it with language modeling. The VQGAN decoder (detokenizer) converts the 256-token sequence back into a coherent, high-resolution image.

How are prompts tokenized and prepared for the encoder?

- Tokenization uses BPE with vocab.json and merges.txt (Hugging Face).
- Special tokens: <s> (start), </s> (end), <pad> (padding).
- Inputs are padded to length 64.
- Two sequences are formed: an “unconditional” row (for classifier-free guidance) and a “conditional” row with the actual prompt. Both are encoded by the BART encoder.

What happens at inference time when generating tokens?

- Start with a BOS placeholder for image tokens.
- For i from 1 to 256, the decoder predicts the next token given: previous image tokens, text embeddings, attention mask, and its cached attention state.
- After 256 tokens are generated, drop the BOS and pass the 256 tokens to the VQGAN detokenizer to get the image.

What do temperature, top-K, and the supercondition factor control?

- Temperature: randomness in sampling (lower ≈ sharper/more conservative; higher ≈ more diverse/creative).
- Top-K: restricts choices to the K most likely tokens at each step (smaller K yields safer/more coherent outputs).
- Supercondition factor: balances unconditional vs. conditional guidance; higher values make the image adhere more strongly to the prompt.

How can I visualize the model’s “painting” process?

- Save the image tokens after each step (or at milestones like 32, 64, …, 256).
- Detokenize these snapshots to images and arrange them in a 2×4 grid to see the image progressively fill in patch-by-patch.
- Create a GIF from all 256 frames to watch the full autoregressive generation unfold.

Which files do I need, and where do they come from?

- Tokenizer: vocab.json, merges.txt (Hugging Face).
- Models: encoder.pt (BART encoder), decoder.pt (BART decoder), detoker.pt (VQGAN detokenizer).
- Store locally (e.g., files/DALLE/) and add the min-dalle repo to your Python path to import its modules.

What shapes and common checks should I expect (and common gotchas)?

- Typical encoder output: [2, 64, 2048] (two rows: unconditional and conditional; 64 tokens; 2048-dim embeddings).
- Image token sequence length: 257 with BOS included; pass only the last 256 to the detokenizer.
- Always mask pads in attention and set modules to eval() for inference.
- Ensure the min-dalle directory is on sys.path, correct device/dtype (e.g., float16 on CUDA), and sufficient GPU memory. If you see odd results, double-check special tokens, padding, BOS handling, and that all weights downloaded correctly.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more