table of content

Part 1. Introduction to generative AI

1 What is generative AI and why PyTorch?

1.1 Introducing generative AI and PyTorch

1.1.1 What is generative AI?

1.1.2 The Python programming language

1.1.3 Using PyTorch as our AI framework

1.2 GANs

1.2.1 A high-level overview of GANs

1.2.2 An illustrating example: Generating anime faces

1.2.3 Why should you care about GANs?

1.3 Transformers

1.3.1 The attention mechanism

1.3.2 The Transformer architecture

1.3.3 Multimodal Transformers and pretrained LLMs

1.4 Why build generative models from scratch?

Summary

2 Deep learning with PyTorch

2.1 Data types in PyTorch

2.1.1 Creating PyTorch tensors

2.1.2 Index and slice PyTorch tensors

2.1.3 PyTorch tensor shapes

2.1.4 Mathematical operations on PyTorch tensors

2.2 An end-to-end deep learning project with PyTorch

2.2.1 Deep learning in PyTorch: A high-level overview

2.2.2 Preprocessing data

2.3 Binary classification

2.3.1 Creating batches

2.3.2 Building and training a binary classification model

2.3.3 Testing the binary classification model

2.4 Multicategory classification

2.4.1 Validation set and early stopping

2.4.2 Building and training a multicategory classification model

Summary

3 Generative adversarial networks: Shape and number generation

3.1 Steps involved in training GANs

3.2 Preparing training data

3.2.1 A training dataset that forms an exponential growth curve

3.2.2 Preparing the training dataset

3.3 Creating GANs

3.3.1 The discriminator network

3.3.2 The generator network

3.3.3 Loss functions, optimizers, and early stopping

3.4 Training and using GANs for shape generation

3.4.1 The training of GANs

3.4.2 Saving and using the trained generator

3.5 Generating numbers with patterns

3.5.1 What are one-hot variables?

3.5.2 GANs to generate numbers with patterns

3.5.3 Training the GANs to generate numbers with patterns

3.5.4 Saving and using the trained model

Summary

Part 2. Image generation

4 Image generation with generative adversarial networks

4.1 GANs to generate grayscale images of clothing items

4.1.1 Training samples and the discriminator

4.1.2 A generator to create grayscale images

4.1.3 Training GANs to generate images of clothing items

4.2 Convolutional layers

4.2.1 How do convolutional operations work?

4.2.2 How do stride and padding affect convolutional operations?

4.3 Transposed convolution and batch normalization

4.3.1 How do transposed convolutional layers work?

4.3.2 Batch normalization

4.4 Color images of anime faces

4.4.1 Downloading anime faces

4.4.2 Channels-first color images in PyTorch

4.5 Deep convolutional GAN

4.5.1 Building a DCGAN

4.5.2 Training and using DCGAN

Summary

5 Selecting characteristics in generated images

5.1 The eyeglasses dataset

5.1.1 Downloading the eyeglasses dataset

5.1.2 Visualizing images in the eyeglasses dataset

5.2 cGAN and Wasserstein distance

5.2.1 WGAN with gradient penalty

5.2.2 cGANs

5.3 Create a cGAN

5.3.1 A critic in cGAN

5.3.2 A generator in cGAN

5.3.3 Weight initialization and the gradient penalty function

5.4 Training the cGAN

5.4.1 Adding labels to inputs

5.4.2 Training the cGAN

5.5 Selecting characteristics in generated images

5.5.1 Selecting images with or without eyeglasses

5.5.2 Vector arithmetic in latent space

5.5.3 Selecting two characteristics simultaneously

Summary

6 CycleGAN: Converting blond hair to black hair

6.1 CycleGAN and cycle consistency loss

6.1.1 What is CycleGAN?

6.1.2 Cycle consistency loss

6.2 The celebrity faces dataset

6.2.1 Downloading the celebrity faces dataset

6.2.2 Process the black and blond hair image data

6.3 Building a CycleGAN model

6.3.1 Creating two discriminators

6.3.2 Creating two generators

6.4 Using CycleGAN to translate between black and blond hair

6.4.1 Training a CycleGAN to translate between black and blond hair

6.4.2 Round-trip conversions of black hair images and blond hair images

Summary

7 Image generation with variational autoencoders

7.1 An overview of AEs

7.1.1 What is an AE?

7.1.2 Steps in building and training an AE

7.2 Building and training an AE to generate digits

7.2.1 Gathering handwritten digits

7.2.2 Building and training an AE

7.2.3 Saving and using the trained AE

7.3 What are VAEs?

7.3.1 Differences between AEs and VAEs

7.3.2 The blueprint to train a VAE to generate human face images

7.4 A VAE to generate human face images

7.4.1 Building a VAE

7.4.2 Training the VAE

7.4.3 Generating images with the trained VAE

7.4.4 Encoding arithmetic with the trained VAE

Summary

Part 3. Natural language processing and Transformers

8 Text generation with recurrent neural networks

8.1 Introduction to RNNs

8.1.1 Challenges in generating text

8.1.2 How do RNNs work?

8.1.3 Steps in training a LSTM model

8.2 Fundamentals of NLP

8.2.1 Different tokenization methods

8.2.2 Word embedding

8.3 Preparing data to train the LSTM model

8.3.1 Downloading and cleaning up the text

8.3.2 Creating batches of training data

8.4 Building and training the LSTM model

8.4.1 Building an LSTM model

8.4.2 Training the LSTM model

8.5 Generating text with the trained LSTM model

8.5.1 Generating text by predicting the next token

8.5.2 Temperature and top-K sampling in text generation

Summary

9 A line-by-line implementation of attention and Transformer

9.1 Introduction to attention and Transformer

9.1.1 The attention mechanism

9.1.2 The Transformer architecture

9.1.3 Different types of Transformers

9.2 Building an encoder

9.2.1 The attention mechanism

9.2.2 Creating an encoder

9.3 Building an encoder-decoder Transformer

9.3.1 Creating a decoder layer

9.3.2 Creating an encoder-decoder Transformer

9.4 Putting all the pieces together

9.4.1 Defining a generator

9.4.2 Creating a model to translate between two languages

Summary

10 Training a Transformer to translate English to French

10.1 Subword tokenization

10.1.1 Tokenizing English and French phrases

10.1.2 Sequence padding and batch creation

10.2 Word embedding and positional encoding

10.2.1 Word embedding

10.2.2 Positional encoding

10.3 Training the Transformer for English-to-French translation

10.3.1 Loss function and the optimizer

10.3.2 The training loop

10.4 Translating English to French with the trained model

Summary

11 Building a generative pretrained Transformer from scratch

11.1 GPT-2 architecture and causal self-attention

11.1.1 The architecture of GPT-2

11.1.2 Word embedding and positional encoding in GPT-2

11.1.3 Causal self-attention in GPT-2

11.2 Building GPT-2XL from scratch

11.2.1 BPE tokenization

11.2.2 The Gaussian error linear unit activation function

11.2.3 Causal self-attention

11.2.4 Constructing the GPT-2XL model

11.3 Loading up pretrained weights and generating text

11.3.1 Loading up pretrained parameters in GPT-2XL

11.3.2 Defining a generate() function to produce text

11.3.3 Text generation with GPT-2XL

Summary

12 Training a Transformer to generate text

12.1 Building and training a GPT from scratch

12.1.1 The architecture of a GPT to generate text

12.1.2 The training process of the GPT model to generate text

12.2 Tokenizing text of Hemingway novels

12.2.1 Tokenizing the text

12.2.2 Creating batches for training

12.3 Building a GPT to generate text

12.3.1 Model hyperparameters

12.3.2 Modeling the causal self-attention mechanism

12.3.3 Building the GPT model

12.4 Training the GPT model to generate text

12.4.1 Training the GPT model

12.4.2 A function to generate text

12.4.3 Text generation with different versions of the trained model

Summary

Part 4. Applications and new developments

13 Music generation with MuseGAN

13.1 Digital music representation

13.1.1 Musical notes, octave, and pitch

13.1.2 An introduction to multitrack music

13.1.3 Digitally represent music: Piano rolls

13.2 A blueprint for music generation

13.2.1 Constructing music with chords, style, melody, and groove

13.2.2 A blueprint to train a MuseGAN

13.3 Preparing the training data for MuseGAN

13.3.1 Downloading the training data

13.3.2 Converting multidimensional objects to music pieces

13.4 Building a MuseGAN

13.4.1 A critic in MuseGAN

13.4.2 A generator in MuseGAN

13.4.3 Optimizers and the loss function

13.5 Training the MuseGAN to generate music

13.5.1 Training the MuseGAN

13.5.2 Generating music with the trained MuseGAN

Summary

14 Building and training a music Transformer

14.1 Introduction to the music Transformer

14.1.1 Performance-based music representation

14.1.2 The music Transformer architecture

14.1.3 Training the music Transformer

14.2 Tokenizing music pieces

14.2.1 Downloading training data

14.2.2 Tokenizing MIDI files

14.2.3 Preparing the training data

14.3 Building a GPT to generate music

14.3.1 Hyperparameters in the music Transformer

14.3.2 Building a music Transformer

14.4 Training and using the music Transformer

14.4.1 Training the music Transformer

14.4.2 Music generation with the trained Transformer

Summary

15 Diffusion models and text-to-image Transformers

15.1 Introduction to denoising diffusion models

15.1.1 The forward diffusion process

15.1.2 Using the U-Net model to denoise images

15.1.3 A blueprint to train the denoising U-Net model

15.2 Preparing the training data

15.2.1 Flower images as the training data

15.2.2 Visualizing the forward diffusion process

15.3 Building a denoising U-Net model

15.3.1 The attention mechanism in the denoising U-Net model

15.3.2 The denoising U-Net model

15.4 Training and using the denoising U-Net model

15.4.1 Training the denoising U-Net model

15.4.2 Using the trained model to generate flower images

15.5 Text-to-image Transformers

15.5.1 CLIP: A multimodal Transformer

15.5.2 Text-to-image generation with DALL-E 2

Summary

16 Pretrained large language models and the LangChain library

16.1 Content generation with the OpenAI API

16.1.1 Text generation tasks with OpenAI API

16.1.2 Code generation with OpenAI API

16.1.3 Image generation with OpenAI DALL-E 2

16.1.4 Speech generation with OpenAI API

16.2 Introduction to LangChain

16.2.1 The need for the LangChain library

16.2.2 Using the OpenAI API in LangChain

16.2.3 Zero-shot, one-shot, and few-shot prompting

16.3 A zero-shot know-it-all agent in LangChain

16.3.1 Applying for a Wolfram Alpha API Key

16.3.2 Creating an agent in LangChain

16.3.3 Adding tools by using OpenAI GPTs

16.3.4 Adding tools to generate code and images

16.4 Limitations and ethical concerns of LLMs

16.4.1 Limitations of LLMs

16.4.2 Ethical concerns for LLMs

Summary

Appendixes

Appendix A: Installing Python, Jupyter Notebook, and PyTorch

A.1 Installing Python and setting up a virtual environment

A.1.1 Installing Anaconda

A.1.2 Setting up a Python virtual environment

A.1.3 Installing Jupyter Notebook

A.2 Installing PyTorch

A.2.1 Installing PyTorch without CUDA

A.2.2 Installing PyTorch with CUDA

Appendix B: Minimally qualified readers and deep learning basics

B.1 Deep learning and deep neural networks

B.1.1 Anatomy of a neural network

B.1.2 Different types of layers in neural networks

B.1.3 Activation Functions

B.2 Training a deep neural network

B.2.1 The training process

B.2.2 Loss functions

B.2.3 Optimizers

Overview

Appendix A. Installing Python, Jupyter Notebook, and PyTorch

This appendix explains how to prepare a reliable, isolated development setup for the book’s projects. It adopts Anaconda as the primary tool for Python distribution, package management, and environment isolation, emphasizing its ease of use and broad compatibility with libraries that can otherwise be difficult to install. The chapter then guides you through installing Jupyter Notebook as the working interface and concludes with installing PyTorch in configurations suited to your hardware, whether CPU-only or GPU-accelerated.

The instructions begin with installing Anaconda on Windows, macOS, or Linux and verifying the installation with Anaconda Navigator. You then create and activate a dedicated virtual environment (for example, named dgai) to keep the book’s dependencies separate from other projects, minimizing conflicts. Within this environment, you install Jupyter Notebook and launch it to ensure everything is ready for running the book’s examples.

Next, the appendix covers installing PyTorch. It shows how to check for a CUDA-enabled GPU (available on Windows and Linux) and provides guidance for both CPU-only and CUDA-enabled setups, including installing Torchvision and Torchaudio alongside PyTorch. It notes that Apple silicon users on recent macOS versions can leverage the MPS backend for GPU acceleration. Finally, it recommends verifying the installation by importing the libraries, printing their versions, and confirming whether the runtime is using the CPU or CUDA.

Figure A.1 Checking if your computer has a CUDA-enabled GPU.

Figure A.2 The interactive interface on how to install PyTorch

FAQ

What is Anaconda and why is it recommended in this book?

Anaconda is an open-source Python distribution that includes a package manager (conda) and environment management. It simplifies installing and managing libraries (via both conda and pip) and makes it easy to isolate project dependencies in separate virtual environments.

How do I install Anaconda on Windows, macOS, or Linux?

Go to https://www.anaconda.com/download/success and download the latest Python 3 installer for your OS.
Windows: Run the graphical installer and follow prompts.
macOS: Use the graphical installer (or command-line installer) and follow prompts.
Linux: Download the bash installer, run it in a terminal, then activate with source ~/.bashrc.

How can I verify that Anaconda installed correctly?

Windows/macOS: Search for and launch “Anaconda Navigator”.
Linux: In a terminal, run anaconda-navigator. If it opens, Anaconda is installed.

How do I create and activate the book’s virtual environment?

In an Anaconda Prompt (Windows) or terminal (macOS/Linux):

conda create -n dgai
conda activate dgai

Press y when prompted during creation. You should see (dgai) in your prompt after activation.

Why use a separate virtual environment for this book?

The environment isolates the packages used in this book from other projects, preventing version conflicts and unintended interference.

How do I install and launch Jupyter Notebook in the dgai environment?

conda activate dgai
conda install notebook
jupyter notebook

Jupyter Notebook will open in your default browser.

How do I check if I have a CUDA‑enabled GPU and find its CUDA version?

CUDA is available on Windows and Linux (not on macOS). In Windows PowerShell or a Linux terminal, run:

nvidia-smi

If you see GPU details, note the CUDA version shown. If you get an error, you don’t have a CUDA‑enabled GPU.

How do I install PyTorch, Torchvision, and Torchaudio without CUDA?

conda activate dgai
conda install pytorch torchvision torchaudio cpuonly -c pytorch

Verify the install:

import torch, torchvision, torchaudio
print(torch.__version__)
print(torchvision.__version__)
print(torchaudio.__version__)

How do I install PyTorch with CUDA using conda?

Determine your CUDA version via nvidia-smi.
Visit https://pytorch.org/get-started/locally/, select your OS, Package = Conda, Language = Python, and the matching CUDA version.
Activate the environment and run the suggested command (example for CUDA 11.8):

conda activate dgai
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

Verify and check device:

import torch, torchvision, torchaudio
print(torch.__version__)
print(torchvision.__version__)
print(torchaudio.__version__)
print("cuda" if torch.cuda.is_available() else "cpu")

My CUDA version isn’t exactly listed (e.g., I have 12.4). What should I select?

Select the closest supported CUDA option on the PyTorch site (e.g., choose CUDA 12.1 for CUDA 12.4). This commonly works and is suggested in the appendix.

I’m using a Mac. Can I use GPU acceleration?

CUDA isn’t available on macOS. If your Mac has Apple silicon or an AMD GPU on macOS 12.3+, you may be able to use PyTorch’s MPS backend for acceleration. See: https://developer.apple.com/metal/pytorch/ and https://pytorch.org/get-started/locally/.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

eBook

pdf, ePub, online

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more