8 Introduction to Deep Learning

Roadmap for this chapter, starting with the high-level implementation of a PyTorch model and followed by detailed coverage of the underlying mechanics that govern our implementation.

Map of human protein kinases, i.e., the Manning kinome tree, clustered based on sequence similarity into eight major groups [3].

Outline the topics we’ll cover as we implement, train, and evaluate our first PyTorch model.

Relationship between the predicted pIC50 values of the test set against their true pIC50 values.

Enrichment plot for our model. The closer the model’s curve to the optimal enrichment line, the better the model’s performance.

Example of a computational graph at a high level of abstraction.

Computational graph for logistic regression. The input node represents the input data fed into the graph. Learnable parameter nodes for the weights and bias are used in conjunction with the input data for intermediate computations, ending with a prediction, 𝑎, that is compared to the true class label, 𝑦, to compute the loss.

Prior to backpropagation, our model makes a prediction, and this prediction is compared to the actual target to compete the error. To figure out which parts of the graph contributed most to the error, we start from the end (the error) and work backwards through the solution process. The error is propagated backwards through the network and the contribution of each parameter (weight and bias) to the error is calculated. These parameters are adjusted slightly to reduce the error.

We generalize the computational graph from figure 8.7 to support an arbitrary number of inputs. We also simplify the computational graph’s representation of the trainable parameters.

A neural network can be composed of artificial neurons across an arbitrary number of hidden layers (i.e., network depth) and number of neurons per layer (i.e., network width). Each neuron can be decomposed into a computational graph similar to what we visualized in previous figures, which we then chain to form the larger, more complex network.

Comparison of activation functions: sigmoid (i.e., logistic), hyperbolic tangent, rectified linear unit (ReLU), and step function. In the earlier computational graphs of a logistic classifier, we used the sigmoid or logistic activation function. In our modeling approach, we switched to using the ReLU activation function.

In this example, a dropout rate of 0.5 has been set, resulting in 50% of hidden neurons being dropped in a given training iteration.

Logging the train and validation loss with each epoch of training. The validation loss saturates before 50 epochs, which informs further hyperparameter tweaking (e.g., cut down the number of epochs or use early stopping, adjust the learning rate or scheduler, etc.).

• Deep learning is a powerful tool in various chemical domains, including drug discovery, materials science, and quantum chemistry. By leveraging neural networks capable of learning complex patterns from vast amounts of data, deep learning algorithms uncover insights that were previously unattainable through traditional computational methods.
• Protein kinases are one of the main protein families targeted in drug design, especially for cancer and inflammatory diseases. As of 2024, there are around 80 approved small molecule protein kinase inhibitors, with an additional 150 in clinical testing. Despite this success, the targets of the FDA-approved kinase inhibitors represent a small percentage of the entire kinase protein family – a large portion of the kinome is underexplored. From the perspective of “the glass Is half full”, the underexplored kinome presents many opportunities for development of novel drugs.
• Tensors are the fundamental data structure in PyTorch. They are multi-dimensional arrays similar to NumPy arrays but with additional features that make them suitable for deep learning computations. Tensors can be created on CPU or GPU (for accelerated computations), support a wide range of mathematical operations, and encode the inputs, outputs, and parameters of a model.
• Our basic PyTorch workflow involved preparing the data by converting inputs into numerical representations to be ingested by our neural network as tensors and preparing the model architecture by specifying its structure in PyTorch. We then fit the model to the data, evaluated the model’s predictions on test data against actual values, reviewed modifications to the architecture (e.g., ReLU activation function, Adam optimizer, learning rate scheduling, regularization via dropout) to improve the model, and saved our final model for later reuse.
• A common goal in any ranking or screening process, e.g., virtual screening, is to prioritize the identification of a small number of the most relevant (active) compounds as quickly as possible, rather than screening through the entire data set. Hence, we use the model to score compounds and, ideally, the most likely actives appear near the top of the ranked list. This early recognition problem encapsulates our desire that we find a large portion of active compounds quickly, i.e., after screening only a small percentage of the dataset, to accelerate hit discovery. The enrichment factor (EF) is a key metric in evaluating how well a method satisfies the early recognition problem.
• A computational graph is a representation of a mathematical expressions as a directed graph. In the computational graph, each node represents an operation or variable and each edge represents the flow of data between operations. Computation graphs are fundamental to how modern deep learning frameworks like PyTorch work. They allow for efficient computation and, crucially, for automatic differentiation, which is key to backpropagating gradient computations and training neural networks.
• Our logistic regression model is analogous to a single-layer neural network composed of only a single neuron with a logistic, or sigmoid, activation function. We chain multiple neurons together, with the output of one neuron feeding into a successive neuron, to construct neural networks of arbitrary width and depth for improved feature extraction. The composition of the neural network’s number of layers and neurons is referred to as its architecture. A neural network’s architecture incorporates three types of layers – one input layer with a node for each input, one or more hidden layers full of artificial neurons, and one output layer representing the network’s output.

FAQ