11 VQGAN: Convert images into sequences of integers

This chapter explains how VQGAN turns images into sequences of discrete tokens so they can be modeled like language by transformers. It motivates the need for a discrete, language-like representation of images as a key step in modern text-to-image systems and frames VQGAN as the bridge between continuous visual data and token sequences. The narrative orients readers around the end goal—encoding and decoding images as integer sequences—while situating VQGAN within the broader pipeline that ultimately powers models like DALL-E.

The chapter builds the concepts progressively: from autoencoders to VAEs (probabilistic latents and KL regularization) to VQ-VAEs (discrete codebooks that improve sharpness and efficiency), and finally to VQGAN, a GAN whose generator is a VQ-VAE. Concretely, an image (e.g., 3×384×384) is encoded into a latent grid of size 24×24 with 256 channels, yielding 576 feature vectors. Each vector is quantized by nearest-neighbor lookup in a learned codebook (e.g., 1024 embeddings of dimension 256), producing a sequence of 576 integer indices. VQGAN is trained with a weighted combination of reconstruction (typically LPIPS perceptual loss), adversarial, and quantization losses, achieving high-fidelity reconstructions while preserving a discrete, structured latent space that is ideal for transformer modeling.

Practically, the chapter demonstrates using a pretrained VQGAN to: encode an image into its latent grid, quantize to codebook indices (the integer sequence), and decode either from the quantized latents or directly from the indices back to a high-resolution image. It highlights where minor reconstruction differences arise (quantization approximation) and shows that the integer sequence alone suffices to faithfully regenerate the image. By the end, readers understand both the inner workings and hands-on usage of VQGAN, setting up the next step: training a transformer to predict image tokens conditioned on text, then decoding those tokens with VQGAN to realize text-to-image generation.

Eight steps for building a text-to-image generator from scratch. This chapter focuses on step 7: transforming an image into a sequence of integers using VQGAN. By achieving this, we unlock the ability to generate images sequentially with transformer models, a critical advance that powers state-of-the-art systems like DALL-E.

A pretrained VQGAN model reconstructs an image. The left image shows the original fish image. The VQGAN encoder transforms this image into a sequence of 576 integers, representing its compressed latent code. The VQGAN decoder then reconstructs the image from this sequence, producing the version shown on the right. The reconstructed image closely resembles the original, demonstrating the effectiveness of the VQGAN in capturing and preserving key visual features through discrete latent representations.

The architecture and training workflow of an autoencoder designed to generate high-resolution color images. An autoencoder has two primary components: an encoder (shown on the left) and a decoder (shown on the right). During training, input images are first passed through the encoder, which compresses them into deterministic latent representations (latent vectors). These latent vectors are then passed through the decoder, which reconstructs the original images. During training, the autoencoder continually adjusts its parameters by minimizing the reconstruction loss, calculated as the difference between the original images and their reconstructed versions.

The architecture of a variational autoencoder (VAE) and the training steps involved in generating high-resolution color images. A VAE consists of two primary components: an encoder (upper middle left) and a decoder (lower middle right). During each training iteration, original images are fed into the encoder, which compresses them into probabilistic representations characterized by mean and variance vectors in the latent space. Encodings are then sampled from these distributions and passed to the decoder. The decoder reconstructs images based on these sampled latent vectors. The VAE optimizes its parameters by minimizing a loss function composed of two terms: the reconstruction loss, which quantifies differences between the original and reconstructed images, and the Kullback-Leibler (KL) divergence, which measures how much the encoder’s latent distribution diverges from a predefined standard normal distribution.

The architecture of a VQ-VAE and the training process. A VQ-VAE has three components: an encoder, a codebook, and a decoder. During training, we pass images through the encoder to obtain latent representations, z. We replace the latent vectors in z with the nearest vectors in the codebook, zq. Therefore, the image can be represented by a sequence of integers, each pointing to the index of the feature vector in the codebook. The decoder then reconstructs the image based on the discrete latent representation zq. We train the model to minimize the reconstruction loss and the quantization loss.

VQGAN architecture and training process. VQGAN combines a VQ-VAE generator with a discriminator in a GAN framework. The generator aims to produce images that are visually indistinguishable from real samples in the training set. Meanwhile, the discriminator attempts to determine whether a given image is "real" (from the training data) or "fake" (produced by the generator). During training, the generator reconstructs images from real inputs, producing synthetic ("fake") samples, which are then mixed with real images and fed into the discriminator. The discriminator predicts whether each image is real or fake, and its predictions are compared against the ground truth labels. Based on the results, both the generator and discriminator update their parameters to improve their respective performance, making the generator better at fooling the discriminator and the discriminator better at detecting fakes.

Comparing reconstructed images with originals. A cat image and a fish image are processed and resized to 3x384x384. We then feed the two processed images to the pretrained VQGAN model to obtain two reconstructed images.

Converting an image into a sequence of integers. The VQGAN model first encodes an image into a continuous latent representation, z, in the latent space, which consists of 576 feature vectors. Each of these vectors is matched with one of the 1024 discrete feature vectors in the codebook. Therefore, the image can be represented by a sequence of integers, with each integer representing the index of the feature vector in the codebook.

Reconstructing an image from a sequence of integers. Each integer in the sequence corresponds to the index of a feature vector in the VQGAN codebook, allowing the model to retrieve the associated feature vectors. These 576 vectors collectively form the quantized latent representation, zq. The VQGAN decoder then uses zq to reconstruct the original image.

Comparing the image reconstructed based on quantized vectors with the one reconstructed based on the sequence of integers. The left image is reconstructed based on the quantized latent representation generated by the VQGAN encoder, z_q. The right image is reconstructed based on the sequence of 576 integers. We look up feature vectors in the codebook based on the integers in the sequence. These feature vectors form a quantized latent representation, z_q2, which is then fed to the VQGAN decoder to reconstruct the image.

• Traditional variational autoencoders (VAEs) encode images into probabilistic latent distributions and reconstruct them by sampling from these distributions. This process often leads to blurry outputs due to the averaging effect of sampling.
• Vector-quantized VAEs (VQ-VAEs) improve upon traditional VAEs by replacing continuous latent distributions with discrete codes from a learned codebook. This leads to sharper reconstructions and more interpretable, efficient latent representations.
• Generative adversarial networks (GANs) consist of a generator and a discriminator trained adversarially: the generator learns to create realistic images, while the discriminator learns to distinguish real from fake.
• VQGAN integrates the discrete latent structure of VQ-VAEs with the adversarial training of GANs. This hybrid model benefits from both stability and semantic structure, enabling the generation of high-resolution, perceptually realistic images.
• The VQGAN loss function blends three key components: adversarial loss for realism, perceptual loss (e.g., LPIPS) for visual fidelity, and quantization loss for effective discretization of the latent space.
• VQGAN encodes images as sequences of discrete tokens, making it a foundational component in models like DALL-E, where images are generated from symbolic sequences using transformer architectures.

FAQ