4 Image Generation with GANs

This chapter moves from simple sequence generation to practical image synthesis with GANs, showing how to build, train, and evaluate models in PyTorch for both grayscale and high-resolution color images. A key design principle is to construct the generator by mirroring the discriminator’s structure, ensuring shapes and channels align as data flow is inverted from noise to image. To scale image fidelity and efficiency, the chapter introduces convolutional building blocks, explains how standard convolutions downsample and extract spatial features, and how transposed convolutions upsample to form higher-resolution feature maps. Throughout, model quality is judged primarily by visual inspection, reflecting common practice in GAN training.

The first project targets 28×28 grayscale clothing images using multilayer perceptrons. The discriminator is a binary classifier over flattened inputs, and the generator reverses its layer dimensions, mapping a 100-dimensional latent vector through fully connected layers to a Tanh output in the range [-1, 1]. Training alternates updates to the discriminator on real and fake batches and updates to the generator to fool the discriminator, using binary cross-entropy loss and Adam optimization. Periodic visualization monitors progress as outputs sharpen from noise to recognizable items; after training, the discriminator is discarded and the generator is saved for sampling.

To generate high-resolution color images, the chapter transitions to DCGANs with convolutional layers for downsampling, transposed convolutions for upsampling, and batch normalization to stabilize training. It details how kernel size, stride, and padding control spatial dimensions, and motivates weight sharing and local receptive fields for parameter efficiency. The DCGAN discriminator stacks Conv2d, BatchNorm2d, and LeakyReLU layers to compress 3-channel inputs to a real/fake probability, while the generator mirrors this hierarchy with ConvTranspose2d, BatchNorm2d, and ReLU layers that map 100-dimensional noise to 64×64 RGB outputs with Tanh. Using Adam with tuned betas, the model is trained on channels-first tensors and evaluated visually after each epoch; the trained generator is then saved and used to produce diverse Anime faces, setting the stage for controlled generation in the next chapter.

Designing a generator network to create clothing items by mirroring the layers in the discriminator network. The right side of the diagram shows the discriminator network, which contains four dense layers. To design a generator that can conjure up clothing items from thin air, we mirror the layers in the discriminator network. Specifically, as shown on the left half of the figure, the generator has four similar dense layers in it, but in reverse order: the first layer in the generator mirrors the last layer in the discriminator, the second layer in the generator mirrors the second to last layer in the discriminator, and so on. Further, in each of the top three layers, the numbers of inputs and outputs in the discriminator are reversed and used as the numbers of outputs and inputs in the generator.

Output from the GAN model to generate clothing items before training. Since the model is not trained, the generated images are nothing like the images in the training set.

Clothing items generated by an image GAN model after ten epochs of training

Clothing items generated by a trained image GAN model (after 50 epochs)

A numerical example of how convolutional operations work, with stride equal to 1 and no padding

A numerical example of how transposed convolutional operations work

Examples from the Anime faces training dataset.

Designing a generator network in DCGAN to create Anime faces by mirroring the layers in the discriminator network. The right side of the diagram shows the discriminator network, which contains five 2D convolutional layers. To design a generator that can conjure up Anime faces out of thin air, we mirror the layers in the discriminator network. Specifically, as shown on the left half of the figure, the generator has five 2D transposed convolutional layers in it, symmetric to the 2D convolutional layers in the discriminator. Further, in each of the top four layers, the numbers of input and output channels in the discriminator are reversed and used as the numbers of output and input channels in the generator.

Generated images in DCGAN after one epoch of training.

Generated Anime face images by the trained generator in DCGAN.

• To conjure up realistic-looking images out of thin air, the generator mirrors layers used in the discriminator network.
• While it’s feasible to generate grayscale images by using just fully connected layers, to generate high-resolution color images, we need to use convolutional neural networks (CNNs).
• 2D convolutional layers are used for feature extraction. They apply a set of learnable filters (also known as kernels) to the input data to detect patterns and features at different spatial scales. These layers are essential for capturing hierarchical representations of the input data.
• 2D Transposed Convolutional Layers (also known as Deconvolution or Upsampling Layers) are used for upsampling or generating high-resolution feature maps. They apply a filter to the input data. However, unlike standard convolution, they increase the spatial dimensions by inserting gaps between the output values, which effectively "upscales" the feature maps. This process generates feature maps of a higher resolution.
• 2D Batch Normalization is a technique commonly used in deep learning and neural networks to improve the training and performance of convolutional neural networks (CNNs) and other models that work with 2D data, such as images. It normalizes the values for each feature channel, so they have a mean of 0 and a standard deviation of 1, which helps stabilize and speed up training.

FAQ