table of content

Part 1. Introduction to generative AI

1 What is generative AI and why PyTorch?

1.1 Introducing generative AI and PyTorch

1.1.1 What is generative AI?

1.1.2 The Python programming language

1.1.3 Using PyTorch as our AI framework

1.2 GANs

1.2.1 A high-level overview of GANs

1.2.2 An illustrating example: Generating anime faces

1.2.3 Why should you care about GANs?

1.3 Transformers

1.3.1 The attention mechanism

1.3.2 The Transformer architecture

1.3.3 Multimodal Transformers and pretrained LLMs

1.4 Why build generative models from scratch?

Summary

2 Deep learning with PyTorch

2.1 Data types in PyTorch

2.1.1 Creating PyTorch tensors

2.1.2 Index and slice PyTorch tensors

2.1.3 PyTorch tensor shapes

2.1.4 Mathematical operations on PyTorch tensors

2.2 An end-to-end deep learning project with PyTorch

2.2.1 Deep learning in PyTorch: A high-level overview

2.2.2 Preprocessing data

2.3 Binary classification

2.3.1 Creating batches

2.3.2 Building and training a binary classification model

2.3.3 Testing the binary classification model

2.4 Multicategory classification

2.4.1 Validation set and early stopping

2.4.2 Building and training a multicategory classification model

Summary

3 Generative adversarial networks: Shape and number generation

3.1 Steps involved in training GANs

3.2 Preparing training data

3.2.1 A training dataset that forms an exponential growth curve

3.2.2 Preparing the training dataset

3.3 Creating GANs

3.3.1 The discriminator network

3.3.2 The generator network

3.3.3 Loss functions, optimizers, and early stopping

3.4 Training and using GANs for shape generation

3.4.1 The training of GANs

3.4.2 Saving and using the trained generator

3.5 Generating numbers with patterns

3.5.1 What are one-hot variables?

3.5.2 GANs to generate numbers with patterns

3.5.3 Training the GANs to generate numbers with patterns

3.5.4 Saving and using the trained model

Summary

Part 2. Image generation

4 Image generation with generative adversarial networks

4.1 GANs to generate grayscale images of clothing items

4.1.1 Training samples and the discriminator

4.1.2 A generator to create grayscale images

4.1.3 Training GANs to generate images of clothing items

4.2 Convolutional layers

4.2.1 How do convolutional operations work?

4.2.2 How do stride and padding affect convolutional operations?

4.3 Transposed convolution and batch normalization

4.3.1 How do transposed convolutional layers work?

4.3.2 Batch normalization

4.4 Color images of anime faces

4.4.1 Downloading anime faces

4.4.2 Channels-first color images in PyTorch

4.5 Deep convolutional GAN

4.5.1 Building a DCGAN

4.5.2 Training and using DCGAN

Summary

5 Selecting characteristics in generated images

5.1 The eyeglasses dataset

5.1.1 Downloading the eyeglasses dataset

5.1.2 Visualizing images in the eyeglasses dataset

5.2 cGAN and Wasserstein distance

5.2.1 WGAN with gradient penalty

5.2.2 cGANs

5.3 Create a cGAN

5.3.1 A critic in cGAN

5.3.2 A generator in cGAN

5.3.3 Weight initialization and the gradient penalty function

5.4 Training the cGAN

5.4.1 Adding labels to inputs

5.4.2 Training the cGAN

5.5 Selecting characteristics in generated images

5.5.1 Selecting images with or without eyeglasses

5.5.2 Vector arithmetic in latent space

5.5.3 Selecting two characteristics simultaneously

Summary

6 CycleGAN: Converting blond hair to black hair

6.1 CycleGAN and cycle consistency loss

6.1.1 What is CycleGAN?

6.1.2 Cycle consistency loss

6.2 The celebrity faces dataset

6.2.1 Downloading the celebrity faces dataset

6.2.2 Process the black and blond hair image data

6.3 Building a CycleGAN model

6.3.1 Creating two discriminators

6.3.2 Creating two generators

6.4 Using CycleGAN to translate between black and blond hair

6.4.1 Training a CycleGAN to translate between black and blond hair

6.4.2 Round-trip conversions of black hair images and blond hair images

Summary

7 Image generation with variational autoencoders

7.1 An overview of AEs

7.1.1 What is an AE?

7.1.2 Steps in building and training an AE

7.2 Building and training an AE to generate digits

7.2.1 Gathering handwritten digits

7.2.2 Building and training an AE

7.2.3 Saving and using the trained AE

7.3 What are VAEs?

7.3.1 Differences between AEs and VAEs

7.3.2 The blueprint to train a VAE to generate human face images

7.4 A VAE to generate human face images

7.4.1 Building a VAE

7.4.2 Training the VAE

7.4.3 Generating images with the trained VAE

7.4.4 Encoding arithmetic with the trained VAE

Summary

Part 3. Natural language processing and Transformers

8 Text generation with recurrent neural networks

8.1 Introduction to RNNs

8.1.1 Challenges in generating text

8.1.2 How do RNNs work?

8.1.3 Steps in training a LSTM model

8.2 Fundamentals of NLP

8.2.1 Different tokenization methods

8.2.2 Word embedding

8.3 Preparing data to train the LSTM model

8.3.1 Downloading and cleaning up the text

8.3.2 Creating batches of training data

8.4 Building and training the LSTM model

8.4.1 Building an LSTM model

8.4.2 Training the LSTM model

8.5 Generating text with the trained LSTM model

8.5.1 Generating text by predicting the next token

8.5.2 Temperature and top-K sampling in text generation

Summary

9 A line-by-line implementation of attention and Transformer

9.1 Introduction to attention and Transformer

9.1.1 The attention mechanism

9.1.2 The Transformer architecture

9.1.3 Different types of Transformers

9.2 Building an encoder

9.2.1 The attention mechanism

9.2.2 Creating an encoder

9.3 Building an encoder-decoder Transformer

9.3.1 Creating a decoder layer

9.3.2 Creating an encoder-decoder Transformer

9.4 Putting all the pieces together

9.4.1 Defining a generator

9.4.2 Creating a model to translate between two languages

Summary

10 Training a Transformer to translate English to French

10.1 Subword tokenization

10.1.1 Tokenizing English and French phrases

10.1.2 Sequence padding and batch creation

10.2 Word embedding and positional encoding

10.2.1 Word embedding

10.2.2 Positional encoding

10.3 Training the Transformer for English-to-French translation

10.3.1 Loss function and the optimizer

10.3.2 The training loop

10.4 Translating English to French with the trained model

Summary

11 Building a generative pretrained Transformer from scratch

11.1 GPT-2 architecture and causal self-attention

11.1.1 The architecture of GPT-2

11.1.2 Word embedding and positional encoding in GPT-2

11.1.3 Causal self-attention in GPT-2

11.2 Building GPT-2XL from scratch

11.2.1 BPE tokenization

11.2.2 The Gaussian error linear unit activation function

11.2.3 Causal self-attention

11.2.4 Constructing the GPT-2XL model

11.3 Loading up pretrained weights and generating text

11.3.1 Loading up pretrained parameters in GPT-2XL

11.3.2 Defining a generate() function to produce text

11.3.3 Text generation with GPT-2XL

Summary

12 Training a Transformer to generate text

12.1 Building and training a GPT from scratch

12.1.1 The architecture of a GPT to generate text

12.1.2 The training process of the GPT model to generate text

12.2 Tokenizing text of Hemingway novels

12.2.1 Tokenizing the text

12.2.2 Creating batches for training

12.3 Building a GPT to generate text

12.3.1 Model hyperparameters

12.3.2 Modeling the causal self-attention mechanism

12.3.3 Building the GPT model

12.4 Training the GPT model to generate text

12.4.1 Training the GPT model

12.4.2 A function to generate text

12.4.3 Text generation with different versions of the trained model

Summary

Part 4. Applications and new developments

13 Music generation with MuseGAN

13.1 Digital music representation

13.1.1 Musical notes, octave, and pitch

13.1.2 An introduction to multitrack music

13.1.3 Digitally represent music: Piano rolls

13.2 A blueprint for music generation

13.2.1 Constructing music with chords, style, melody, and groove

13.2.2 A blueprint to train a MuseGAN

13.3 Preparing the training data for MuseGAN

13.3.1 Downloading the training data

13.3.2 Converting multidimensional objects to music pieces

13.4 Building a MuseGAN

13.4.1 A critic in MuseGAN

13.4.2 A generator in MuseGAN

13.4.3 Optimizers and the loss function

13.5 Training the MuseGAN to generate music

13.5.1 Training the MuseGAN

13.5.2 Generating music with the trained MuseGAN

Summary

14 Building and training a music Transformer

14.1 Introduction to the music Transformer

14.1.1 Performance-based music representation

14.1.2 The music Transformer architecture

14.1.3 Training the music Transformer

14.2 Tokenizing music pieces

14.2.1 Downloading training data

14.2.2 Tokenizing MIDI files

14.2.3 Preparing the training data

14.3 Building a GPT to generate music

14.3.1 Hyperparameters in the music Transformer

14.3.2 Building a music Transformer

14.4 Training and using the music Transformer

14.4.1 Training the music Transformer

14.4.2 Music generation with the trained Transformer

Summary

15 Diffusion models and text-to-image Transformers

15.1 Introduction to denoising diffusion models

15.1.1 The forward diffusion process

15.1.2 Using the U-Net model to denoise images

15.1.3 A blueprint to train the denoising U-Net model

15.2 Preparing the training data

15.2.1 Flower images as the training data

15.2.2 Visualizing the forward diffusion process

15.3 Building a denoising U-Net model

15.3.1 The attention mechanism in the denoising U-Net model

15.3.2 The denoising U-Net model

15.4 Training and using the denoising U-Net model

15.4.1 Training the denoising U-Net model

15.4.2 Using the trained model to generate flower images

15.5 Text-to-image Transformers

15.5.1 CLIP: A multimodal Transformer

15.5.2 Text-to-image generation with DALL-E 2

Summary

16 Pretrained large language models and the LangChain library

16.1 Content generation with the OpenAI API

16.1.1 Text generation tasks with OpenAI API

16.1.2 Code generation with OpenAI API

16.1.3 Image generation with OpenAI DALL-E 2

16.1.4 Speech generation with OpenAI API

16.2 Introduction to LangChain

16.2.1 The need for the LangChain library

16.2.2 Using the OpenAI API in LangChain

16.2.3 Zero-shot, one-shot, and few-shot prompting

16.3 A zero-shot know-it-all agent in LangChain

16.3.1 Applying for a Wolfram Alpha API Key

16.3.2 Creating an agent in LangChain

16.3.3 Adding tools by using OpenAI GPTs

16.3.4 Adding tools to generate code and images

16.4 Limitations and ethical concerns of LLMs

16.4.1 Limitations of LLMs

16.4.2 Ethical concerns for LLMs

Summary

Appendixes

Appendix A: Installing Python, Jupyter Notebook, and PyTorch

A.1 Installing Python and setting up a virtual environment

A.1.1 Installing Anaconda

A.1.2 Setting up a Python virtual environment

A.1.3 Installing Jupyter Notebook

A.2 Installing PyTorch

A.2.1 Installing PyTorch without CUDA

A.2.2 Installing PyTorch with CUDA

Appendix B: Minimally qualified readers and deep learning basics

B.1 Deep learning and deep neural networks

B.1.1 Anatomy of a neural network

B.1.2 Different types of layers in neural networks

B.1.3 Activation Functions

B.2 Training a deep neural network

B.2.1 The training process

B.2.2 Loss functions

B.2.3 Optimizers

Overview

16 Pre-Trained LLMs and the LangChain Library

Pre-trained large language models have reshaped generative AI by enabling high-quality text, code, image, and speech generation without training models from scratch. This chapter explains how to harness these capabilities effectively, emphasizing Python-based integration for automation and customization and introducing LangChain to orchestrate multiple models and services. It also frames prompting styles—zero-shot, one-shot, and few-shot—as core techniques for guiding model behavior, and closes with a balanced discussion of limitations and ethical concerns.

Practically, the chapter demonstrates using the OpenAI API from Python for tasks such as question answering, summarization, creative writing, code generation, image synthesis, and text-to-speech. Because LLMs are trained on past data and lack real-time awareness, the chapter shows how LangChain can build an agent that selects the right tool for a query, chaining LLMs with external services like Wolfram Alpha for computations and current data and Wikipedia for factual lookups. With LangChain’s agent and tool abstractions (including ReAct-style reasoning), readers progressively add custom tools—a text summarizer, joke teller, sentiment classifier, and generators for code and images—so the agent can autonomously decide how to solve diverse tasks.

Finally, the chapter highlights critical caveats: LLMs can make reasoning errors, hallucinate facts, and fail on nuanced problems; they also raise concerns around bias, misinformation, privacy, and copyright. It recommends mitigation through careful prompt design, tool augmentation for verification and retrieval, bias-aware development practices, robust content moderation, and transparent, responsible deployment. The result is a practical blueprint for building capable, extensible LLM-powered systems while maintaining rigor and ethical safeguards.

Use OpenAI GPT-3.5 to generate Python code to plot a sine curve. We use the text description “Write a Python program to plot a sine graph” to ask it to generate a Python program. We then run the program to create the graph.

An image generated by DALL-E 2 with the text prompt “someone fishing at the riverbank.”

Steps to create a zero-shot know-it-all agent with the LangChain library

Applying for a Wolfram Alpha AppID

Add a tool in LangChain to generate Python code. The tool then generates code to plot sine and cosine curves in the same graph, with a legend and line styles.

An image generated by a know-it-all agent in LangChain.

How GPT-4 still makes low-level mistakes.

Summary

Few-shot prompting means you give LLMs multiple examples to help them understand the task, while one-shot or zero-shot prompting means one example or no example is provided.
LangChain is a Python library designed to facilitate the use of LLMs in various applications. It abstracts away the complexities of interacting with different LLMs and applications. It allows the agent to automatically go to the right tool in the toolbox based on the task at hand without explicitly telling it what to do.
Modern pre-trained LLMs such as OpenAI’s GPT series can create various formats of content such as text, images, audio, and code.
Despite their impressive achievements, LLMs lack a true understanding of the content or the ability to reason. These limitations can lead to errors in logic, factual inaccuracies, and a failure to grasp complex concepts or nuances. Furthermore, the rapid advancement and widespread application of these models have given rise to a host of ethical concerns such as bias, misinformation, breach of privacy, and copyright infringements. These issues demand careful consideration and proactive measures to ensure that the development and deployment of LLMs align with ethical standards and societal values.

FAQ

How do I set up my environment to follow this chapter?

Install the required packages, then add your API keys. In a fresh environment or notebook, run: pip install --upgrade openai langchain_openai langchain, pip install wolframalpha langchainhub, and pip install --upgrade --quiet wikipedia. You need an OpenAI API key (from the prior chapter) and, for Wolfram Alpha, an AppID set as the environment variable WOLFRAM_ALPHA_APPID. Using Jupyter Notebook is recommended for running the examples.

What kinds of content can I generate with pre-trained LLMs via the OpenAI API?

You can generate text (Q&A, summaries, creative writing), code (e.g., Python), images (with DALL-E 2), and audio (text-to-speech). Text/code use chat.completions with messages that include roles: system (behavior), user (instructions), and assistant (model replies). Image generation uses the images API, and speech uses the audio.speech API.

What’s the difference between zero-shot, one-shot, and few-shot prompting?

Zero-shot gives only instructions with no examples; the model relies on prior knowledge. One-shot provides a single example to illustrate the task. Few-shot supplies multiple examples so the model can infer the pattern (useful for tasks like classification). Keep your example format consistent (e.g., using the same separators) so the model understands the mapping.

Why use Python and LangChain instead of a browser UI?

Python enables automation, customization, and integration with systems and data pipelines (e.g., scheduled reports, conditional logic, error handling). LangChain adds orchestration: it abstracts model/tool specifics, lets you chain LLMs with external tools (e.g., Wikipedia, Wolfram Alpha), and compose multi-step agents for more accurate, task-appropriate solutions.

Can GPT still generate code, and what happened to Codex?

Yes. Code generation is handled within GPT-3.5/GPT-4 chat endpoints; Codex’s capabilities are integrated rather than exposed as a separate model. Provide a clear natural-language description (requirements, libraries, constraints) and the model will produce runnable code (e.g., Matplotlib plots in Python).

How do I generate images with DALL-E 2, and what should I expect?

Call the images API (model “dall-e-2”) with a text prompt, size, and count. The response includes a temporary URL (expires in about an hour). Image outputs are stochastic: the same prompt can yield different images across runs due to the underlying diffusion process.

What is LangChain, and what problems does it solve?

LangChain is a framework for building LLM-powered applications that coordinate models and tools. It simplifies calling different LLMs/APIs, manages prompts, and enables agents that decide which tool to use for a task, such as combining GPT with Wolfram Alpha for real-time data and Wikipedia for recent facts.

What is a ReAct agent in LangChain and how does it work?

A ReAct agent (e.g., the “hwchase17/react” prompt) interleaves reasoning with tool use. Given a question and a toolbox (Wolfram Alpha, Wikipedia, custom tools), it decides which tool to call, observes the result, reasons about next steps, and produces a final answer. The AgentExecutor can show its step-by-step reasoning and tool calls when verbose mode is on.

How do I add Wolfram Alpha and Wikipedia so the agent can answer real-time or recent questions?

Obtain a Wolfram Alpha AppID and set WOLFRAM_ALPHA_APPID. In LangChain, load tools like “wolfram-alpha” and “wikipedia” with your LLM (e.g., ChatOpenAI). The agent will route queries: Wolfram Alpha for computations and real-time info; Wikipedia for factual lookups about recent or specific events.

How do I create custom tools in LangChain (summarizer, joke teller, code/image generators)?

Define a PromptTemplate and an LLMChain for each task, then wrap it as a Tool (Tool.from_function) with a clear name and description. Recreate the agent with the expanded tool list. Phrase inputs to match the tool template so the agent selects the right tool. For images, you can pair a text-to-image prompt chain with DALL·E via DallEAPIWrapper.

What limitations and ethical concerns should I keep in mind?

LLMs can make logical or factual errors, and they lack true understanding. They may miss real-time events due to training cutoffs, and can exhibit bias or produce misinformation. Privacy and copyright concerns arise from training data and outputs. Mitigations include tool integration for up-to-date facts (e.g., Wolfram/Wikipedia), careful prompt design, bias detection/correction, content moderation, and adherence to evolving legal/ethical guidelines.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

eBook

pdf, ePub, online

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more