table of content

Part 1 Understanding attention and transformers

1 A tale of two models: Transformers and diffusions

1.1 What is a text-to-image generation model?

1.1.1 Unimodal vs. multimodal models

1.1.2 Practical use cases of text-to-image models

1.2 Transformer-based text-to-image generation

1.2.1 Converting an image into a sequence of integers and then back

1.2.2 Training and using a transformer-based text-to-image model

1.3 Text-to-image generation with diffusion models

1.3.1 Forward and reverse diffusions

1.3.2 Latent diffusion models and Stable Diffusion

1.4 How to build text-to-image models from scratch

1.5 Challenges for text-to-image generation models

1.5.1 Are generative AI models stealing from artists?

1.5.2 The geometric inconsistency problem

1.6 Social, environmental, and ethical concerns

2 Build a transformer

2.1 An overview of attention and transformers

2.1.1 How the attention mechanism works

2.1.2 How to create a transformer

2.2 Word embedding and positional encoding

2.2.1 Word tokenization with the Spacy library

2.2.2 A sequence padding function

2.2.3 Input embedding from word embedding and positional encoding

2.3 Creating an encoder–decoder transformer

2.3.1 Coding the attention mechanism

2.3.2 Defining the Transformer() class

2.3.3 Creating a language translator

2.4 Training and using the German-to-English translator

2.4.1 Training the encoder–decoder transformer

2.4.2 Translating German to English with the trained model

3 Classify images with a vision transformer

3.1 The blueprint to train a ViT

3.1.1 Converting images to sequences

3.1.2 Training a ViT for classification

3.2 The CIFAR-10 dataset

3.2.1 Downloading and visualizing CIFAR-10 images

3.2.2 Preparing datasets for training and testing

3.3 Building a ViT from scratch

3.3.1 Dividing images into patches

3.3.2 Modeling the positions of different patches in an image

3.3.3 Using the multi-head self-attention mechanism

3.3.4 Building an encoder-only transformer

3.3.5 Using the ViT to create a classifier

3.4 Training and using the ViT to classify images

3.4.1 Choosing the optimizer and the loss function

3.4.2 Training the ViT for image classification

3.4.3 Classifying images using the trained ViT

4 Add captions to images

4.1 Training and using a transformer to add captions

4.1.1 Preparing data and the causal attention mask

4.1.2 Creating and training a transformer

4.2 Preparing the training dataset

4.2.1 Downloading and visualizing Flickr 8k images

4.2.2 Building a vocabulary of tokens

4.2.3 Preparing the training dataset

4.3 Creating a multimodal transformer to add captions

4.3.1 Defining a ViT as the image encoder

4.3.2 Creating the decoder to generate text

4.4 Training and using the image-to-text transformer

4.4.1 Training the encoder–decoder transformer

4.4.2 Adding captions to images with the trained model

Part 2 Introduction to diffusion models

5 Generate images with diffusion models

5.1 The forward diffusion process

5.1.1 How diffusion models work

5.1.2 Visualizing the forward diffusion process

5.1.3 Different diffusion schedules

5.2 The reverse diffusion process

5.3 A blueprint to train the U-Net model

5.3.1 Steps in training a denoising U-Net model

5.3.2 Preprocessing the training data

5.4 Training and using the diffusion model

5.4.1 The Denoising Diffusion Probabilistic Model noise scheduler

5.4.2 Inference using the U-Net denoising model

5.4.3 Training and using the denoising U-Net model

6 Control what images to generate in diffusion models

6.1 Classifier-free guidance in diffusion models

6.1.1 An overview of classifier-free guidance

6.1.2 A blueprint to implement CFG

6.2 Different components of a denoising U-Net model

6.2.1 Time step embedding and label embedding

6.2.2 The U-Net denoising model architecture

6.2.3 Down blocks and up blocks in the U-Net

6.3 Building and training the denoising U-Net model

6.3.1 Building the denoising U-Net

6.3.2 The Denoising Diffusion Probabilistic Model

6.3.3 Training the diffusion model

6.4 Generating images with the trained diffusion model

6.4.1 Visualizing generated images

6.4.2 How the guidance parameter affects generated images

7 Generate high-resolution images with diffusion models

7.1 Attention in U-Net, DDIM, and image interpolation

7.1.1 Incorporating the attention mechanism in the U-Net model

7.1.2 Denoising Diffusion Implicit Models

7.1.3 Image interpolation in diffusion models

7.2 High-resolution flower images as training data

7.2.1 Visualizing images in the training dataset

7.2.2 Applying forward diffusion on flower images

7.3 Building and training a U-Net for high-resolution images

7.3.1 Building the denoising U-Net model

7.3.2 Training the denoising U-Net model

7.4 Image generation and interpolation

7.4.1 Using the trained denoising U-Net to generate images

7.4.2 Transition from one image to another

Part 3 Text-to-image generation with diffusion models

8 CLIP: A model to measure the similarity between image and text

8.1 The CLIP model

8.1.1 How the CLIP model works

8.1.2 Selecting an image from Flickr 8k based on a text description

8.2 Preparing the training dataset

8.2.1 Image-caption pairs in Flickr 8k

8.2.2 The DistilBERT tokenizer

8.2.3 Preprocess captions and images for training

8.3 Creating a CLIP model

8.3.1 Creating a text encoder

8.3.2 Creating an image encoder

8.3.3 Building a CLIP model

8.4 Training and using the CLIP model

8.4.1 Training the CLIP model

8.4.2 Using the trained CLIP model to select images

8.4.3 Using the OpenAI pretrained CLIP model to select images

9 Text-to-image generation with latent diffusion

9.1 What is a latent diffusion model?

9.1.1 How variational autoencoders work

9.1.2 Combining a latent diffusion model with a variational autoencoder

9.2 Compressing and reconstructing images with VAEs

9.2.1 Downloading the pretrained VAE

9.2.2 Encoding and decoding images with the pretrained VAE

9.3 Text-to-image generation with latent diffusion

9.3.1 Guidance by the CLIP model

9.3.2 Diffusion in the latent space

9.3.3 Converting latent images to high-resolution ones

9.4 Modifying existing images with text prompts

10 A deep dive into Stable Diffusion

10.1 Generating images with Stable Diffusion

10.2 The Stable Diffusion architecture

10.2.1 Generating images from text with Stable Diffusion

10.2.2 Text embedding interpolation

10.3 Creating text embeddings

10.4 Image generation in the latent space

10.5 Converting latent images to high-resolution ones

Part 4 Text-to-image generation with transformers

11 VQGAN: Convert images into sequences of integers

11.1 Converting images into sequences of integers and back

11.2 Variational autoencoders

11.2.1 What is an autoencoder?

11.2.2 The need for VAEs and their training methodology

11.3 Vector quantized variational autoencoders

11.3.1 The need for VQ-VAEs

11.3.2 The VQ-VAE model architecture and training process

11.4 Vector quantized generative adversarial networks

11.4.1 Generative adversarial networks

11.4.2 VQGAN: A GAN with a VQ-VAE generator

11.5 A pretrained VQGAN model

11.5.1 Reconstructing images with the pretrained VQGAN

11.5.2 Converting images into sequences of integers

12 A minimal implementation of DALL-E

12.1 How min-DALL-E works

12.1.1 Training min-DALL-E

12.1.2 From prompt to pixels: Image generation at inference time

12.2 Tokenizing and encoding the text prompt

12.2.1 Tokenizing the text prompt

12.2.2 Encoding the text prompt

12.3 Iterative prediction of image tokens

12.3.1 Loading the pretrained BART decoder

12.3.2 Predicting image tokens using the BART decoder

12.4 Converting image tokens to high-resolution images

12.4.1 Loading the pretrained VQGAN detokenizer

12.4.2 Visualizing the intermediate and final high-resolution outputs

Part 5 New developments and challenges

13 New developments and challenges in text-to-image generation

13.1 State-of-the-art text-to-image generators

13.1.1 DALL-E series

13.1.2 Google’s Imagen

13.1.3 Latent diffusion models: Stable Diffusion and Midjourney

13.2 Challenges and concerns

13.3 A blueprint to fine-tune ResNet50

13.3.1 The history and architecture of ResNet50

13.3.2 A plan to fine-tune ResNet50 for classification

13.3.3 Using ResNet50 to classify images

13.4 Fine-tuning ResNet50 to detect fake images

13.4.1 Downloading and preprocessing real and fake face images

13.4.2 Fine-tuning ResNet50

13.4.3 Detecting deepfakes using the fine-tuned ResNet50

Appendix

Installing PyTorch and enabling GPU training locally and in Colab

A.1 Installing Python and setting up a virtual environment

A.1.1 Installing Anaconda

A.1.2 Setting up a Python virtual environment

A.1.3 Installing Jupyter Notebook

A.2 Installing PyTorch

A.3 Using Google Colab for GPU training and inference

references

Overview

13 New developments and challenges in text-to-image generation

The chapter offers a concise tour of how modern text-to-image systems convert language into compelling visuals, tracing the arc from early transformer-based tokenization to today’s diffusion-driven approaches. It highlights the DALL·E series and its improvements in text-image alignment and instruction following, Google’s Imagen with T5-XXL text encoders and cascaded diffusion for progressive refinement, and latent diffusion models such as Stable Diffusion and MidJourney, which generate in a compressed latent space to lower compute while empowering iterative, collaborative creative workflows.

It then examines the field’s core challenges and societal risks. Technically, models still struggle with geometric consistency due to weak 3D priors and limited understanding of physics, while training and deployment demand significant computation with environmental costs. Societally, misuse—especially deepfakes and misinformation—raises ethical and legal concerns, alongside unresolved questions about copyright and fair use in web-scale training data. The chapter also underscores the importance of bias auditing and mitigation, warning that both under- and over-correction can misrepresent people and contexts. Progress, it argues, requires better alignment, robust evaluation, and multidisciplinary governance.

To pair innovation with safeguards, the chapter presents a practical blueprint for detecting AI-generated images by fine-tuning ResNet-50. Instead of training from scratch, it builds a balanced real-versus-fake dataset, applies the pretrained model’s transforms, replaces the classification head with a two-class output, freezes the backbone, and trains the new head with early stopping before testing on unseen images. The resulting detector attains strong accuracy, offering a resource-efficient approach to deepfake identification and equipping readers with a dual perspective: how to create powerful generators and how to evaluate and defend against the risks they introduce.

The architecture of ResNet-50. ResNet-50 is structured as follows: (1) An initial convolutional layer followed by max pooling to reduce spatial dimensions; (2) Four sequential “residual blocks” composed of bottleneck units with residual connections; (3) An average pooling layer; (4) A fully connected output layer. The model takes an image as input and outputs logits for 1,000 object classes, which can be converted into predicted probabilities using the softmax activation function.

Training from scratch vs. fine-tuning an existing model. Training from scratch (top row in the diagram) involves building a model with randomly initialized parameters, requiring the network to learn all features from raw data. In contrast, fine-tuning uses the architecture and pretrained weights of an existing model, modifying (often only) the output layer for a new downstream task. The bulk of the network retains the learned parameters, while only the final layers are updated for the new objective.

Modified ResNet-50 architecture for deepfake detection. All original layers of ResNet-50 are retained, except for the last fully connected output layer. The original output head (with 1,000 output channels) is replaced with a new head (with 2 output channels), producing logits for the “real” and “fake” classes. Parameters in the new output layer are initialized randomly and trained on our dataset. All other parameters are inherited from the pretrained model and kept frozen during fine-tuning.

Using ResNet-50 to make predictions on images. Each image is shown with its top predicted class and the corresponding probability as determined by ResNet-50. The results highlight both the strengths and quirks of large-scale classification models: for example, the model may predict “bee eater” for a bird image that looks like a kingfisher, since “kingfisher” isn’t among the 1,000 ImageNet classes. Such observations show that the model’s predictions depend not only on the input image, but also on the set of categories it was trained to recognize.

Sample real and fake images from the 140k Real and Fake Faces dataset. The top row displays five real face images, while the bottom row shows five face images generated by AI. At a glance, both sets look convincingly realistic. This side-by-side comparison highlights the subtlety of the deepfake detection challenge: to the naked eye, even AI-generated images can appear authentic. Such visualizations emphasize the need for advanced machine learning methods to distinguish between real and fake images.

Using the fine-tuned model to differentiate real images from deepfakes. Each image is labeled with both its ground-truth class (real or fake) and the model’s prediction. This side-by-side visualization provides immediate feedback on the model’s predictions. Notice how convincingly realistic some “fake” images appear, underscoring the importance of robust deep learning solutions.

Summary

The DALL-E series harness the combined strengths of transformer architectures and advanced image generation techniques. Each successive version of DALL-E has pushed the boundaries of what’s achievable in generating images from textual descriptions.
Imagen combines pretrained language models with advanced cascaded diffusion architectures, generating remarkably precise, high-resolution images directly from sophisticated and nuanced textual descriptions.
Latent diffusion models such as Stable Diffusion have emerged as a powerful paradigm in text-to-image generation by substantially reducing computational costs while maintaining visual quality and semantic fidelity. MidJourney extends the core capabilities of Stable Diffusion into user-friendly, creatively empowering applications.
Despite remarkable progress, state-of-the-art text-to-image generation models still have many challenges and concerns. These include geometric inconsistency and the societal risks posed by misinformation, bias, and copyright concerns.
Training from scratch involves building a model with randomly initialized parameters, requiring the network to learn all features from raw data. In contrast, fine-tuning uses the architecture and pretrained weights of an existing model, modifying (often only) the output layer for a downstream task. The bulk of the network retains the learned parameters, while only the final layers are updated for the new objective.
Fine-tuning a neural network requires high-quality, well-labeled data that closely matches your target task. For our deepfake detection problem, this means gathering and organizing a large number of images, clearly labeled as either “real” or “fake.” The more representative and diverse the dataset, the better the model will be at generalizing to new, unseen images.
The fine-tuned ResNet-50 model performs well on unseen, out-of-sample images, achieving an accuracy of 93.73% among a test dataset.

FAQ

How do today’s leading text-to-image systems differ (DALL-E, Imagen, Stable Diffusion, MidJourney)?

DALL-E evolved from a transformer that predicted image tokens (via VQVAE) to DALL-E 2’s CLIP-guided approach for better text-image alignment, and DALL-E 3’s stronger instruction following through improved caption-aware training. Google’s Imagen encodes text with T5-XXL and uses a cascaded diffusion pipeline that refines from 64x64 up to 1024x1024 for exceptional photorealism. Stable Diffusion operates in a compressed latent space (latent diffusion), cutting compute while retaining quality. MidJourney builds on the latent diffusion paradigm but focuses on user experience: prompt guidance, interactive refinement, collaboration, and rich stylistic controls.

What is the “geometric inconsistency” problem, and why do models struggle with it?

Most models are trained on 2D images without explicit 3D structure or physical constraints, so they learn correlations rather than true 3D reasoning. As a result, perspective, depth, lighting, and object relationships can be inconsistent across a scene or viewpoints. Improving geometric consistency likely requires 3D-aware priors (e.g., multi-view supervision), depth or normal cues, or explicit physics/geometry components that enforce spatial coherence.

What are the computational and environmental concerns with state-of-the-art models?

Training and serving large text-to-image models require substantial compute and energy, raising cost and sustainability questions. Mitigations include: - Using more efficient architectures (e.g., latent diffusion) - Transfer learning and fine-tuning instead of training from scratch - Pruning, quantization, and other compression techniques - Careful inference optimizations and hardware-aware deployment The goal is to reduce footprint while maintaining quality.

How can text-to-image models be misused, and what safeguards does the chapter explore?

They can be used to create deepfakes and misleading content that undermines trust and public discourse. The chapter emphasizes responsible use, content moderation, and collaboration across developers, policymakers, and media. Practically, it walks through fine-tuning a ResNet-50 classifier to distinguish real from AI-generated images as a defensive measure.

What legal and copyright issues arise from training on web-scraped data?

Training data often includes copyrighted material obtained without explicit consent, prompting debates over ownership and fair use. Supporters argue training is transformative; critics argue it exceeds fair use due to scale and direct ingestion. The chapter notes the issue remains unresolved and likely requires legislative and judicial guidance to balance the interests of creators, users, and developers.

Why fine-tune ResNet-50 for deepfake detection instead of training from scratch?

Fine-tuning leverages rich visual features learned from large datasets like ImageNet, drastically reducing data, time, and compute needs. The approach in the chapter: - Load a pretrained ResNet-50 - Replace the final fully connected layer with a 2-class head (real vs fake) - Freeze the rest of the network - Train only the new head on a real/fake dataset This yields strong performance with minimal resources.

Which dataset and preprocessing pipeline are used in the chapter’s detector?

The 140k Real and Fake Faces dataset from Kaggle (balanced: 70k real, 70k fake) with train/validation/test splits. Images are resized and normalized to ResNet-50’s expected input (3×224×224) using torchvision’s default ResNet-50 transforms. DataLoaders batch and shuffle samples for efficient training and evaluation.

What is the training setup (loss, optimizer, regularization, early stopping)?

- Loss: CrossEntropyLoss for two classes - Optimizer: Adam (lr≈1e-3) - Strategy: Freeze all pretrained layers; train only the new head - Regularization: Dropout in the new head; validation monitoring - Early stopping: Stop if validation loss doesn’t improve for 3 consecutive epochs; keep best weights This balances speed, stability, and generalization.

How well did the fine-tuned model perform, and how should results be evaluated?

On the provided test set (balanced real/fake), the fine-tuned ResNet-50 achieved roughly 94% accuracy in the chapter’s run. Accuracy is appropriate for balanced data, but in other settings consider confusion matrices, precision/recall, F1, ROC–AUC, and calibration. It’s also important to test robustness to new generators and distribution shifts, not just the dataset used for fine-tuning.

What remaining challenges should practitioners keep in mind?

- Bias: Models can reflect or amplify social biases; auditing and mitigation are essential, while avoiding overcorrection that distorts facts. - Detection arms race: As generators improve, detectors must be updated and validated on new data. - Geometry and realism: Better 3D/physics priors are needed for consistent scenes. - Governance: Clear policies, consent practices, and responsible-release norms remain crucial as capabilities advance.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more