5 Selecting Characteristics in Generated Images

This chapter shows how to steer generative image models so outputs exhibit desired attributes, moving beyond unconstrained results from earlier GANs. It introduces two complementary control mechanisms: conditioning on labels (conditional GANs) and selecting or interpolating latent vectors that correlate with visual traits. To make training stable and improve fidelity—especially at higher resolutions—the chapter integrates Wasserstein distance with a gradient penalty (WGAN-GP), yielding smoother optimization, mitigating mode collapse, and producing sharper, more realistic faces.

The workflow begins with a curated eyeglasses dataset, where images are organized and labels verified to support supervised conditioning. A conditional GAN is built in PyTorch with a generator that consumes a 100-dimensional noise vector concatenated with a one-hot label, and a critic (WGAN terminology for discriminator) that scores images while also receiving the label as extra channels alongside RGB. Training uses careful weight initialization, Adam optimization, and the WGAN-GP objective, which enforces a 1‑Lipschitz critic via a gradient penalty computed on interpolations between real and generated samples. Images are normalized and resized to 256×256 to target higher quality than earlier chapters, and progress is monitored by periodically sampling the generator under different label settings.

After training, attribute selection becomes practical and flexible. Label control directly toggles features such as the presence or absence of eyeglasses, while latent vector selection highlights instance-specific traits (for example, vectors that tend to yield more masculine or feminine facial characteristics). Interpolating labels produces smooth transitions in a single attribute (label arithmetic), and interpolating latent vectors morphs identity or style (vector arithmetic). Combining both lets the model specify two independent attributes at once—such as facial characteristics and eyeglasses—delivering four targeted groups in one framework. The chapter closes by weighing trade-offs: conditional GANs require labeled data but produce diverse samples per attribute, whereas handpicked latent vectors need no labels but tend to control only individual instances unless many vectors are curated.

Sample images with eyeglasses in the training dataset.

Sample images without eyeglasses in the training dataset.

A diagram of Wasserstein GAN with gradient penalty. The discriminator network in Wasserstein GAN (which we call the critic) rates input images: it tries to assign a score of -∞ to a fake image (bottom left) and a score of ∞ to the real image (top middle). Further, an interpolated image of the real and fake images (top left) is presented to the critic, and the gradient penalty with respect to the critic’s rating on the interpolated image is added to the total loss in the training process.

A diagram of the training process for conditional GANs.

Images of human faces with eyeglasses that are generated by the trained conditional GAN model.

Images of human faces without eyeglasses that are generated by the trained conditional GAN model.

Label arithmetic in conditional GAN. We first create two labels: the no eyeglasses label labels_ng and the eyeglasses label labels_g. These two labels instruct the trained generator to produce images with and without eyeglasses in them, respectively. We then create five interpolated labels, each as a weighted average of the original two labels: w*labels_ng +(1-w)*labels_g, where the weight w takes five different values, 0, 0.25, 0.5, 0.75, and 1. The five generated images based on the five interpolated labels are shown above. The image on the far left has eyeglasses. As we move from the left to the right, the eyeglasses gradually fade away, until the image on the far right has no eyeglasses in it.

Vector arithmetic in GAN. We first save two random noise vectors z_female_ng and z_male_ng. The two vectors lead to images of female and male faces, respectively. We then create five interpolated vectors, each as a weighted average of the original two vectors: w*z_female_ng+(1-w)*z_male_ng, where the weight w takes five different values, 0, 0.25, 0.5, 0.75, and 1. The five generated images based on the five interpolated vectors are shown above. The image on the far left has male features. As we move from the left to the right, the male features gradually fade away and the female features gradually appear, until the image on the far right shows a female face in it.

Select two characteristics simultaneously in the generated image. We select a noise vector from the following two choices z_female_ng and z_male_ng. We also select a label from the following two choices: labels_ng and labels_g. We then feed the noise vector and the label to the trained generator to create an image. Based on the values of the noise vector and the label, the trained model can create four types of images. By doing this, we effectively select two independent characteristics in the generated image: a male or a female face, and whether the image has eyeglasses in it or not.

Conduct vector arithmetic and label arithmetic simultaneously. The value of i changes from 0 to 35; p and q are the integer quotient and remainder, respectively, when i is divided by 6. Therefore, p and q each can take six different values, 0, 1, 2, 3, 4, and 5. The interpolated noise vector, z_female_ng*p/5+z_male_ng*(1-p/5), and the interpolated label, labels_ng[0]*q/5+labels_g[0]*(1-q/5), can each take six different values. In each row, when you go from left to right, the eyeglasses gradually fade away. In each column, when you go from top to bottom, the image changes gradually from a male face to a female face.

• By selecting a certain noise vector in the latent space and feeding it to the trained GAN model, we can select a certain characteristic in the generated image, such as whether the generated image has a male or female face in it.
• A conditional GAN is different from a traditional GAN. We train the model on labeled data and ask the trained model to generate data with a specific attribute. For example, one label tells the model to generate images of human faces with eyeglasses while another tells the model to create human faces without eyeglasses.
• After a conditional GAN is trained, we can use a series of weighted averages of the labels to generate images that transition from an image represented by one label to an image represented by another label. For example, a series of images in which the eyeglasses gradually fade away on the same person’s face. We call this label arithmetic.
• We can also use a series of weighted averages of two different noise vectors to create images that transition from one attribute to another. For example, a series of images in which the male features gradually fade away and female features gradually appear. We call this vector arithmetic.
• Wasserstein GAN (WGAN) is a technique used to improve the training stability and performance of GAN models by using Wasserstein distance instead of the binary cross-entropy as the loss function. Further, for the Wasserstein distance to work correctly, the critic in WGANs must be 1-Lipschitz continuous, meaning the gradient norms of the critic's function must be at most 1 everywhere. The gradient penalty in WGANs adds a regularization term to the loss function to enforce the Lipschitz constraint more effectively.

FAQ