10 Generative Models for De Novo Design

This chapter presents de novo molecular design as a way to confront the immense challenge of chemical space, where the number of possible drug-like compounds is so vast that brute-force search is hopeless. Traditional approaches such as high-throughput screening and rational design remain useful, but they are slow, expensive, and limited by available libraries and iterative design cycles. Generative models offer a different strategy: instead of modifying molecules atom by atom, they learn patterns from data and create new compounds from scratch, aiming to produce candidates with desirable properties such as potency, solubility, and safety.

The chapter first builds intuition for why a good molecular latent space matters. A simple autoencoder can compress SMILES strings into latent vectors and reconstruct them, but by itself it often learns an unstructured space of isolated points that does not support meaningful generation. To make the space useful, it must be valid, reconstructive, and continuous, so that nearby points correspond to similar molecules and random samples decode into chemically sensible structures. The text explains tokenization, embeddings, encoder-decoder design, weight initialization, and evaluation metrics, then shows that a basic autoencoder can reconstruct molecules well but fails at true generation because random latent samples usually produce invalid outputs.

To solve this, the chapter introduces variational autoencoders, recurrent neural networks, and GRU-based sequence models, culminating in a VAE-CYC style architecture. The VAE regularizes latent space with a KL term so sampled points are drawn from a smooth, continuous distribution, while cyclical annealing helps prevent posterior collapse by repeatedly encouraging the model to use the latent code. RNNs and especially GRUs are used because SMILES strings are sequential and contain long-range dependencies such as ring closures and branching. The chapter also improves the input representation with SmilesPE, a chemistry-aware byte-pair encoding scheme that shortens sequences and strengthens inductive bias, resulting in a more structured latent space that is better suited for generating novel molecules and serving as a foundation for later reinforcement learning optimization.

Overview of targeted molecular generation with latent reinforcement learning. The system begins with an input molecule that is encoded into a compressed latent vector representation. An RL agent operates within this latent space, proposing optimization actions to navigate toward regions associated with desired molecular properties (such as "Active" and "Non-toxic" regions shown). The decoder then reconstructs these optimized latent vectors back into novel molecular structures. Rewards are calculated based on the properties of the generated molecules and fed back to improve the RL agent's decision-making. This approach enables continuous optimization in latent space rather than discrete structural modifications, allowing guided generation of molecules with specific target properties.

Core components of an autoencoder neural network applied to molecular structures. The process flows from left to right: an input molecule (X) is fed into an encoder network that compresses the molecular representation into a low-dimensional latent vector. This compressed representation captures the most essential features of the molecule in a compact form. The decoder network then attempts to reconstruct the original molecule (X') from this latent vector. The reconstruction loss (shown by the dashed red arrow) measures the difference between the input molecule and the reconstructed output, providing the training signal that forces the network to learn meaningful molecular representations. The information bottleneck created by the latent vector ensures the autoencoder must learn to encode only the most important molecular features necessary for accurate reconstruction.

Three-step process of converting molecular SMILES strings into dense vector representations for neural network processing. (1) Character-level tokenization breaks the input sequence into individual tokens, including special tokens like <sos> (start of sequence) and <eos> (end of sequence). (2) Vocabulary mapping converts each token to its corresponding integer ID using a learnable lookup table. (3) The embedding matrix (30 tokens × 128 dimensions) transforms these integer IDs into dense, learnable vector representations. The bottom flow diagram shows how this embedding layer fits into the overall autoencoder pipeline, taking a molecular input and converting the tokenized text to numerical vectors before passing to the encoder for compression. This approach enables the model to learn semantic relationships between chemically similar tokens during training, providing a more efficient and meaningful representation than sparse one-hot encoding.

Unrolling an RNN for sequence processing. The left side shows the conceptual "rolled" representation of an RNN processing an input molecule. The right side shows the "unrolled" version that illustrates how the RNN processes the sequence. At each time step, the same neural network (purple circles) takes the current input token and the hidden state from the previous step (h_{t-1}) to produce an output (y_t) and an updated hidden state (h_t) that carries forward to the next step. This sequential processing allows the network to build up understanding of the molecular structure by maintaining memory of previously processed tokens through the evolving hidden state. The bias term is omitted for clarity.

Dissection of the simple RNN cell, as specified in listing 10.7. Note that the output layer computation is not visualized, but would be calculated using the hidden state, ℎ𝑡, as follows: 𝑦𝑡 = 𝑊ℎ𝑦ℎ𝑡 + 𝑏𝑦

Internal architecture of a Gated Recurrent Unit (GRU) cell. The diagram illustrates the information flow through a GRU cell, showing the two key gates (reset gate rt and update gate zt) that control memory retention and updates. The reset gate determines how much previous hidden state information to retain when computing the candidate hidden state h̃t, while the update gate controls the blend between the previous hidden state ht-1 and the new candidate state to produce the final output ht. The mathematical equations on the right define the computations for each component, with σ representing the sigmoid activation function and ⊗ denoting elementwise multiplication.

Architecture of a bidirectional RNN for SMILES string encoding. The network processes the input sequence (x₀, x₁, ..., x_{T-1}, x_T) in both forward (left-to-right, top row) and backward (right-to-left, bottom row) directions simultaneously. Each time step produces a hidden state that concatenates the forward hidden state and backward hidden state. The bidirectional processing allows the model to capture dependencies flowing in both directions through the molecular sequence, with each token's representation informed by both its predecessors and successors in the SMILES string.

Bidirectional RNN Encoder Architecture for VAE-CYC. The encoder processes input molecules by first converting them from chemical structure to SMILES representation, then through an embedding layer that converts text to numerical vectors. A multi-layer bidirectional RNN encoder processes the sequence in both forward and backward directions. The final hidden states from both directions are concatenated and passed through a linear projection layer to generate latent distribution parameters. These parameters, combined with random noise through the reparameterization trick, produce the final latent vector representation of the input molecule.

The 4-layer unidirectional GRU decoder used in the VAE-CYC implementation for generating SMILES molecular representations. The decoder is initialized with hidden states derived from the latent vector through linear projection. Concatenation of the latent vector with input character embeddings at every timestep resolves the information bottleneck problem inherent in standard seq2seq architectures. During training, ground truth characters are passed to the next layer instead of the decoder's own predictions (teacher forcing). The architecture processes characters sequentially with each GRU layer's output serving as input to the next layer at inference. The final layer outputs logits that are converted to probability distributions via softmax for next-character prediction, reconstructing the molecular structure X'.

• The Challenge of Chemical Space: We establish the problem in de novo molecular design: the vastness of "chemical space" makes traditional search methods inefficient for discovering drugs with multiple desired properties. The proposed solution is to use generative models to learn a simplified, continuous map of this space, known as a latent space, allowing for efficient navigation and design of novel molecules.
• The Basic Autoencoder and Its Limitations: The initial model constructed is a basic autoencoder, which uses an encoder-decoder architecture to compress and reconstruct molecular SMILES strings. While this model achieves high accuracy in reconstructing molecules it has already seen, it fundamentally fails at generating new, valid molecules. Its latent space is unstructured and discontinuous, i.e., a map of "isolated islands," making it impossible to explore for novel chemistry.
• Creating a Continuous Map with Variational Autoencoders (VAEs): To solve the generation problem, we introduce the Variational Autoencoder (VAE). Instead of mapping a molecule to a single point, a VAE maps it to a probability distribution. This is achieved through a probabilistic encoder and a unique loss function that combines the standard reconstruction loss with a Kullback-Leibler (KL) divergence term. This KL term regularizes the latent space, forcing it to be smooth, continuous, and well-organized, which is essential for generating new molecules by sampling.
• Advanced Techniques for a Robust VAE: We detail techniques for making the VAE effective. The reparameterization trick the model to be trained despite the random sampling step. To combat posterior collapse, where the model ignores the latent space, cyclical annealing dynamically adjusts the weight of the KL divergence loss during training, ensuring the model learns a chemically meaningful representation.
• Handling Sequential Data with Recurrent Neural Networks (GRUs): Recognizing that SMILES strings are sequential data where order is critical, the model's architecture is upgraded from simple feedforward networks to Gated Recurrent Units (GRUs). GRUs are a type of RNN with internal memory "gates" that allow them to effectively learn the long-range dependencies required to understand chemical syntax (e.g., matching ring closures). The final model uses a bidirectional GRU in the encoder for a complete contextual understanding and a unidirectional GRU in the decoder for step-by-step generation.
• Improving Inductive Bias with Advanced Tokenization: The final key improvement is shifting from simple character-level tokenization to Smiles Pair Encoding (SPE). SPE learns to represent common, recurring chemical motifs and functional groups as single, meaningful tokens. This provides the model with a stronger inductive bias (a "head start" on chemical knowledge), resulting in shorter sequences, faster training, and a more structured latent space that is less prone to collapse.

FAQ