table of content

Part 1. Introduction to generative AI

1 What is generative AI and why PyTorch?

1.1 Introducing generative AI and PyTorch

1.1.1 What is generative AI?

1.1.2 The Python programming language

1.1.3 Using PyTorch as our AI framework

1.2 GANs

1.2.1 A high-level overview of GANs

1.2.2 An illustrating example: Generating anime faces

1.2.3 Why should you care about GANs?

1.3 Transformers

1.3.1 The attention mechanism

1.3.2 The Transformer architecture

1.3.3 Multimodal Transformers and pretrained LLMs

1.4 Why build generative models from scratch?

Summary

2 Deep learning with PyTorch

2.1 Data types in PyTorch

2.1.1 Creating PyTorch tensors

2.1.2 Index and slice PyTorch tensors

2.1.3 PyTorch tensor shapes

2.1.4 Mathematical operations on PyTorch tensors

2.2 An end-to-end deep learning project with PyTorch

2.2.1 Deep learning in PyTorch: A high-level overview

2.2.2 Preprocessing data

2.3 Binary classification

2.3.1 Creating batches

2.3.2 Building and training a binary classification model

2.3.3 Testing the binary classification model

2.4 Multicategory classification

2.4.1 Validation set and early stopping

2.4.2 Building and training a multicategory classification model

Summary

3 Generative adversarial networks: Shape and number generation

3.1 Steps involved in training GANs

3.2 Preparing training data

3.2.1 A training dataset that forms an exponential growth curve

3.2.2 Preparing the training dataset

3.3 Creating GANs

3.3.1 The discriminator network

3.3.2 The generator network

3.3.3 Loss functions, optimizers, and early stopping

3.4 Training and using GANs for shape generation

3.4.1 The training of GANs

3.4.2 Saving and using the trained generator

3.5 Generating numbers with patterns

3.5.1 What are one-hot variables?

3.5.2 GANs to generate numbers with patterns

3.5.3 Training the GANs to generate numbers with patterns

3.5.4 Saving and using the trained model

Summary

Part 2. Image generation

4 Image generation with generative adversarial networks

4.1 GANs to generate grayscale images of clothing items

4.1.1 Training samples and the discriminator

4.1.2 A generator to create grayscale images

4.1.3 Training GANs to generate images of clothing items

4.2 Convolutional layers

4.2.1 How do convolutional operations work?

4.2.2 How do stride and padding affect convolutional operations?

4.3 Transposed convolution and batch normalization

4.3.1 How do transposed convolutional layers work?

4.3.2 Batch normalization

4.4 Color images of anime faces

4.4.1 Downloading anime faces

4.4.2 Channels-first color images in PyTorch

4.5 Deep convolutional GAN

4.5.1 Building a DCGAN

4.5.2 Training and using DCGAN

Summary

5 Selecting characteristics in generated images

5.1 The eyeglasses dataset

5.1.1 Downloading the eyeglasses dataset

5.1.2 Visualizing images in the eyeglasses dataset

5.2 cGAN and Wasserstein distance

5.2.1 WGAN with gradient penalty

5.2.2 cGANs

5.3 Create a cGAN

5.3.1 A critic in cGAN

5.3.2 A generator in cGAN

5.3.3 Weight initialization and the gradient penalty function

5.4 Training the cGAN

5.4.1 Adding labels to inputs

5.4.2 Training the cGAN

5.5 Selecting characteristics in generated images

5.5.1 Selecting images with or without eyeglasses

5.5.2 Vector arithmetic in latent space

5.5.3 Selecting two characteristics simultaneously

Summary

6 CycleGAN: Converting blond hair to black hair

6.1 CycleGAN and cycle consistency loss

6.1.1 What is CycleGAN?

6.1.2 Cycle consistency loss

6.2 The celebrity faces dataset

6.2.1 Downloading the celebrity faces dataset

6.2.2 Process the black and blond hair image data

6.3 Building a CycleGAN model

6.3.1 Creating two discriminators

6.3.2 Creating two generators

6.4 Using CycleGAN to translate between black and blond hair

6.4.1 Training a CycleGAN to translate between black and blond hair

6.4.2 Round-trip conversions of black hair images and blond hair images

Summary

7 Image generation with variational autoencoders

7.1 An overview of AEs

7.1.1 What is an AE?

7.1.2 Steps in building and training an AE

7.2 Building and training an AE to generate digits

7.2.1 Gathering handwritten digits

7.2.2 Building and training an AE

7.2.3 Saving and using the trained AE

7.3 What are VAEs?

7.3.1 Differences between AEs and VAEs

7.3.2 The blueprint to train a VAE to generate human face images

7.4 A VAE to generate human face images

7.4.1 Building a VAE

7.4.2 Training the VAE

7.4.3 Generating images with the trained VAE

7.4.4 Encoding arithmetic with the trained VAE

Summary

Part 3. Natural language processing and Transformers

8 Text generation with recurrent neural networks

8.1 Introduction to RNNs

8.1.1 Challenges in generating text

8.1.2 How do RNNs work?

8.1.3 Steps in training a LSTM model

8.2 Fundamentals of NLP

8.2.1 Different tokenization methods

8.2.2 Word embedding

8.3 Preparing data to train the LSTM model

8.3.1 Downloading and cleaning up the text

8.3.2 Creating batches of training data

8.4 Building and training the LSTM model

8.4.1 Building an LSTM model

8.4.2 Training the LSTM model

8.5 Generating text with the trained LSTM model

8.5.1 Generating text by predicting the next token

8.5.2 Temperature and top-K sampling in text generation

Summary

9 A line-by-line implementation of attention and Transformer

9.1 Introduction to attention and Transformer

9.1.1 The attention mechanism

9.1.2 The Transformer architecture

9.1.3 Different types of Transformers

9.2 Building an encoder

9.2.1 The attention mechanism

9.2.2 Creating an encoder

9.3 Building an encoder-decoder Transformer

9.3.1 Creating a decoder layer

9.3.2 Creating an encoder-decoder Transformer

9.4 Putting all the pieces together

9.4.1 Defining a generator

9.4.2 Creating a model to translate between two languages

Summary

10 Training a Transformer to translate English to French

10.1 Subword tokenization

10.1.1 Tokenizing English and French phrases

10.1.2 Sequence padding and batch creation

10.2 Word embedding and positional encoding

10.2.1 Word embedding

10.2.2 Positional encoding

10.3 Training the Transformer for English-to-French translation

10.3.1 Loss function and the optimizer

10.3.2 The training loop

10.4 Translating English to French with the trained model

Summary

11 Building a generative pretrained Transformer from scratch

11.1 GPT-2 architecture and causal self-attention

11.1.1 The architecture of GPT-2

11.1.2 Word embedding and positional encoding in GPT-2

11.1.3 Causal self-attention in GPT-2

11.2 Building GPT-2XL from scratch

11.2.1 BPE tokenization

11.2.2 The Gaussian error linear unit activation function

11.2.3 Causal self-attention

11.2.4 Constructing the GPT-2XL model

11.3 Loading up pretrained weights and generating text

11.3.1 Loading up pretrained parameters in GPT-2XL

11.3.2 Defining a generate() function to produce text

11.3.3 Text generation with GPT-2XL

Summary

12 Training a Transformer to generate text

12.1 Building and training a GPT from scratch

12.1.1 The architecture of a GPT to generate text

12.1.2 The training process of the GPT model to generate text

12.2 Tokenizing text of Hemingway novels

12.2.1 Tokenizing the text

12.2.2 Creating batches for training

12.3 Building a GPT to generate text

12.3.1 Model hyperparameters

12.3.2 Modeling the causal self-attention mechanism

12.3.3 Building the GPT model

12.4 Training the GPT model to generate text

12.4.1 Training the GPT model

12.4.2 A function to generate text

12.4.3 Text generation with different versions of the trained model

Summary

Part 4. Applications and new developments

13 Music generation with MuseGAN

13.1 Digital music representation

13.1.1 Musical notes, octave, and pitch

13.1.2 An introduction to multitrack music

13.1.3 Digitally represent music: Piano rolls

13.2 A blueprint for music generation

13.2.1 Constructing music with chords, style, melody, and groove

13.2.2 A blueprint to train a MuseGAN

13.3 Preparing the training data for MuseGAN

13.3.1 Downloading the training data

13.3.2 Converting multidimensional objects to music pieces

13.4 Building a MuseGAN

13.4.1 A critic in MuseGAN

13.4.2 A generator in MuseGAN

13.4.3 Optimizers and the loss function

13.5 Training the MuseGAN to generate music

13.5.1 Training the MuseGAN

13.5.2 Generating music with the trained MuseGAN

Summary

14 Building and training a music Transformer

14.1 Introduction to the music Transformer

14.1.1 Performance-based music representation

14.1.2 The music Transformer architecture

14.1.3 Training the music Transformer

14.2 Tokenizing music pieces

14.2.1 Downloading training data

14.2.2 Tokenizing MIDI files

14.2.3 Preparing the training data

14.3 Building a GPT to generate music

14.3.1 Hyperparameters in the music Transformer

14.3.2 Building a music Transformer

14.4 Training and using the music Transformer

14.4.1 Training the music Transformer

14.4.2 Music generation with the trained Transformer

Summary

15 Diffusion models and text-to-image Transformers

15.1 Introduction to denoising diffusion models

15.1.1 The forward diffusion process

15.1.2 Using the U-Net model to denoise images

15.1.3 A blueprint to train the denoising U-Net model

15.2 Preparing the training data

15.2.1 Flower images as the training data

15.2.2 Visualizing the forward diffusion process

15.3 Building a denoising U-Net model

15.3.1 The attention mechanism in the denoising U-Net model

15.3.2 The denoising U-Net model

15.4 Training and using the denoising U-Net model

15.4.1 Training the denoising U-Net model

15.4.2 Using the trained model to generate flower images

15.5 Text-to-image Transformers

15.5.1 CLIP: A multimodal Transformer

15.5.2 Text-to-image generation with DALL-E 2

Summary

16 Pretrained large language models and the LangChain library

16.1 Content generation with the OpenAI API

16.1.1 Text generation tasks with OpenAI API

16.1.2 Code generation with OpenAI API

16.1.3 Image generation with OpenAI DALL-E 2

16.1.4 Speech generation with OpenAI API

16.2 Introduction to LangChain

16.2.1 The need for the LangChain library

16.2.2 Using the OpenAI API in LangChain

16.2.3 Zero-shot, one-shot, and few-shot prompting

16.3 A zero-shot know-it-all agent in LangChain

16.3.1 Applying for a Wolfram Alpha API Key

16.3.2 Creating an agent in LangChain

16.3.3 Adding tools by using OpenAI GPTs

16.3.4 Adding tools to generate code and images

16.4 Limitations and ethical concerns of LLMs

16.4.1 Limitations of LLMs

16.4.2 Ethical concerns for LLMs

Summary

Appendixes

Appendix A: Installing Python, Jupyter Notebook, and PyTorch

A.1 Installing Python and setting up a virtual environment

A.1.1 Installing Anaconda

A.1.2 Setting up a Python virtual environment

A.1.3 Installing Jupyter Notebook

A.2 Installing PyTorch

A.2.1 Installing PyTorch without CUDA

A.2.2 Installing PyTorch with CUDA

Appendix B: Minimally qualified readers and deep learning basics

B.1 Deep learning and deep neural networks

B.1.1 Anatomy of a neural network

B.1.2 Different types of layers in neural networks

B.1.3 Activation Functions

B.2 Training a deep neural network

B.2.1 The training process

B.2.2 Loss functions

B.2.3 Optimizers

Overview

14 Building and Training a Music Transformer

This chapter reframes AI music creation as a sequence modeling problem and walks through building a GPT-style Transformer that composes piano music from performance data. Starting with MIDI performances, music is represented as a stream of discrete events rather than multi-track grids, enabling the use of text-generation techniques to predict the next event in context. The workflow runs end to end: convert MIDI to events, map events to indexes, train a decoder-only Transformer on long sequences to capture musical dependencies, generate new event sequences conditioned on a short prompt, and finally convert the result back to a playable MIDI file.

A performance-based representation underpins the approach, with four event types—note_on, note_off, time_shift, and velocity—capturing pitch, note duration, rhythmic spacing, and dynamics. Time is discretized in 0.01-second increments (binned values), and velocities are quantized into 32 bins to keep the vocabulary compact and training tractable. All unique events form a vocabulary of 388 tokens, augmented by special end-of-sequence and padding tokens for a total of 390. Training data are built from the MAESTRO piano dataset by tokenizing each piece, mapping tokens to indexes, segmenting into sequences of length 2048, and forming input-output pairs via a one-step right shift to teach next-event prediction.

The model is a decoder-only Transformer with six layers, 512-dimensional embeddings, and eight attention heads, using learned positional embeddings and totaling about 20.16 million parameters. It is trained with Adam and cross-entropy loss (ignoring padding), with gradient clipping for stability, on small batches of long sequences to capture long-range musical structure. For generation, a short prompt (e.g., the first few hundred events) seeds autoregressive sampling; temperature scaling controls creativity and diversity in the output. The system produces coherent, stylistically consistent piano music and illustrates how GPT-style Transformers can be adapted from text to performance-driven music generation.

The architecture of a music Transformer. Music files in MIDI formats are first converted into sequences of musical events. These events are then tokenized and converted into indexes. We organize these indexes into sequences of 2048 elements, and each batch contains 2 such sequences. The input sequence first undergoes word embedding and positional encoding, with the input embedding being the sum of these two components. This input embedding is then processed through six decoder layers, each utilizing self-attention mechanisms to capture the relationships among different musical events in the sequence. After passing through the decoder layers, the output undergoes layer normalization to ensure stability in the training process. It then passes through a linear layer, resulting in an output size of 390, which corresponds to the number of unique tokens in the vocabulary. This final output represents the predicted logits for the next musical event in the sequence.

The training process for a music Transformer to generate music.

Summary

The performance-based representation of music enables us to represent a musical piece as a sequence of notes, which include control messages and velocity values. These notes can be further reduced to four kinds of musical events: note-on, note-off, time-shift, and velocity. Each event type can assume various values. Consequently, we can transform a music piece into a sequence of tokens and then into indexes.
A music Transformer adapts the Transformer architecture, originally designed for natural language processing (NLP) tasks, for music generation. This model is designed to generate sequences of musical notes by learning from a large dataset of existing music. It is trained to predict the next note in a sequence based on previous notes, by recognizing patterns, structures, and relationships among various musical elements in the training data.
Just as in text generation, we can use temperature to regulate the creativity of the generated music.

FAQ

What is a music Transformer and why is a decoder-only architecture used here?

The music Transformer is a GPT-style, decoder-only Transformer that models a piece of music as a sequence of discrete event tokens and predicts the next event given all previous ones. Unlike translation, music generation doesn’t need an encoder for a separate source sequence, so a causal, decoder-only stack with self-attention is sufficient and efficient.

How is music represented before tokenization?

Music is represented in a performance-based manner using MIDI-like control messages: note_on (start a pitch), note_off (stop a pitch), time_shift (elapsed time), and velocity (playing intensity). This captures both the structure and expressiveness needed for generation.

How are time and velocity discretized to control vocabulary size?

- Time is quantized into 0.01-second increments and bucketed into 100 values (0–99), where 0 = 0.01s, 99 = 1.00s; longer gaps are represented by multiple time_shift tokens.
- Velocity (0–127) is binned into 32 buckets (0–31). Separating velocity from note_on drastically reduces the number of unique event types.

What are the token types, their value ranges, and index ranges?

- note_on: values 0–127 → indexes 0–127
- note_off: values 0–127 → indexes 128–255 (128 + value)
- time_shift: values 0–99 → indexes 256–355 (256 + value)
- velocity: values 0–31 → indexes 356–387 (356 + value)
Special tokens: EOS = 388 (end of sequence for y), PAD = 389 (padding). Total vocabulary size = 390.

Why is the maximum sequence length set to 2048 tokens?

Music often contains long-range dependencies. A context window of 2048 tokens lets the model capture long-term relations (e.g., motifs, phrasing, pedal/sustain patterns) while remaining trainable on a single GPU with small batch sizes.

What dataset is used and how is it prepared?

The MAESTRO v2.0.0 piano dataset is used. MIDI files are split into train/validation/test per the provided JSON metadata. Each MIDI is converted to events with processor.py, mapped to indexes, then packed into 2048-length input (x) and target (y) sequences, padding with 389 as needed and marking EOS with 388 for y when sequences are short.

What are the key hyperparameters and model size?

- Layers: 6 decoder blocks
- Heads: 8 attention heads (64 dims each for 512 embedding)
- Embedding: 512 (learned token and learned positional embeddings)
- Vocab size: 390
- Block size: 2048
- Dropout: 0.1 (embedding, attention, residual)
Parameter count: approximately 20.16 million.

How is the model trained?

- Objective: next-token prediction with cross-entropy over 390 classes
- Data: (x, y) where y is x shifted right by 1
- Loss: CrossEntropyLoss(ignore_index=389) to exclude PAD
- Optimizer: Adam (lr=1e-4), gradient norm clipping at 1.0
- Epochs: 100 (example run ~3 hours on a CUDA GPU)

How does music generation work after training?

Provide a prompt (e.g., first ~250 tokens) and iteratively sample the next token from softmax probabilities over logits, optionally scaled by temperature, until a target length is reached (e.g., 1000 tokens). The generated index sequence is then decoded back to a playable MIDI file.

What does the temperature parameter do, and do we need top-k?

Temperature rescales logits before softmax. Lower values (e.g., 0.7) make outputs more conservative and predictable; higher values (e.g., 1.5) increase diversity and risk-taking. The chapter uses temperature alone for simplicity, but you can add top-k or nucleus sampling to further control variety.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

eBook

pdf, ePub, online

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more