1 The Drug Discovery Process

Drug discovery is presented as a long, expensive, and failure-prone search problem at the intersection of chemistry, biology, and regulation. The space of possible drug-like molecules is enormous, the number of relevant biological targets is also large and flexible, and even promising compounds often fail because they are toxic, ineffective in the body, hard to manufacture, or blocked by regulatory requirements. These realities make it clear why computational methods are attractive: they can help researchers narrow the search, prioritize better candidates, and surface problems earlier before costly experiments and trials.

The chapter introduces the core language needed to use machine learning in this setting. It explains how molecules can be represented as SMILES strings or as molecular graphs, and how those representations can be turned into numerical features for models. It then defines machine learning and deep learning as systems that learn patterns from labeled or unlabeled data rather than following hand-coded rules, emphasizing the importance of training data quality, generalization, and avoiding overfitting. A small practical example using RDKit, fingerprints, PCA, and logistic regression shows how molecular data can be explored and classified computationally.

Finally, the chapter surveys the main stages of the discovery and development pipeline and where AI can help. It distinguishes target identification, hit discovery, lead identification, lead optimization, preclinical testing, and clinical development, noting that each step filters compounds more stringently. It also highlights major AI applications such as virtual screening, generative chemistry, retrosynthesis, and protein structure prediction, showing how these tools can reduce experimental burden, propose new molecules, and support decision-making. Throughout, the chapter stresses that AI is a powerful aid, but not a substitute for the biological, chemical, and regulatory constraints that determine whether a drug can ultimately succeed.

Drug discovery can be thought of as a difficult search problem that exists at the intersection of the chemical search space of 1063 drug-like compounds and the biological search space of 105 targets.

Two views of the same decline. The upper panel plots new FDA drug approvals per billion dollars of inflation-adjusted R&D on a logarithmic scale, so a constant exponential rate appears as a straight line; open circles are single years, and the heavy line is a five-year centred mean, drawn dashed before 1970 where the spending series has to be reconstructed. You may have heard of Moore's Law, the observation that the number of transistors on an integrated circuit doubles approximately every two years, so that computing power rises while cost falls. Eroom's Law (Moore spelled backwards) runs the other way: the inflation-adjusted R&D cost of developing a new drug doubles roughly every nine years, which is the same statement as the efficiency plotted here halving every 9.9 years between 1950 and 2010. Scannell et al. attribute that decline to diminishing returns arriving from several directions at once, among them lower risk tolerance by regulatory agencies (the "cautious regulator" problem), a reflex to respond to falling productivity by spending more (the "throw money at it" tendency), and the need to show more than a modest incremental benefit over drugs that already work (the "better than the Beatles" problem) [6]. The lower panel shows where the ratio comes from: between 1950 and 2024 real R&D spending grew about 160-fold while approvals grew about 2.5-fold. After 2010 the fall stops, leaving the 2025 ratio about 3.9 times above the extrapolated pre-2010 trend, a break documented by Ringel et al. [7]. The plot is built from FDA approval counts and PhRMA member R&D deflated by the CPI-U, following the construction in Scannell et al. [6] and extended to 2025.

Example pairs of isomeric SMILES.

Artificial intelligence, ML, and deep learning are all related to each other.

Example drug molecules for each USAN stem classification within our data set.

Chemical space exploration in a reduced, 4-dimensional space.

Decision boundary of our logistic regressor for classifying “-cillin” (left) and “-olol” (right) USAN stems. For each plot, colored samples belong to the positive class and uncolored samples belong to the negative class.

Using AI to guide early prediction and optimization of drug-like molecules, we can broaden the number of considered candidate molecules, identify failures earlier when they are relatively inexpensive, and accelerate delivery of novel therapeutics to the clinic for patient benefit.

In virtual screening, we start with a large, diverse library of compounds that we can filter using a predictive model that has learned to predict what properties each compound has. Our predictive model has learned how to map the chemical space to the functional space. If the compound is predicted to have optimal properties, we carry it over for further experiments. In de novo design, we start with a defined set of property criteria that we can use along with a generative model to generate the structure of our ideal drug candidate. Our generative model knows how to map the functional space to the chemical space.

If we know both the structure of our ligand or compound and the target, we can use structured-based design methods. If we only know the ligand structure, we are restricted to ligand-based design methods. Alternatively, if we only know the target structure, we can use de novo design to guide generation of a suitable drug candidate.

We can breakdown drug design into target identification and validation, hit discovery, hit-to-lead (lead identification), lead optimization, and preclinical development. Once a drug candidate has progressed to the drug development stage, it will need to pass multiple phases of clinical trials testing safety and efficacy prior to submission to and review by the FDA and launch to market.

We can break down the ADMET properties into the following broad descriptions. Absorption refers to the process by which a drug enters the bloodstream from its administration site, such as the gastrointestinal tract for oral drugs or the respiratory system for inhalation drugs. Distribution pertains to the movement of a drug within the body once it has entered the bloodstream. Metabolism refers to the biochemical transformation of a drug within the body, primarily carried out by enzymes. Metabolic processes aim to convert drugs into more polar and water-soluble metabolites, facilitating their elimination from the body. Excretion involves the removal of drugs and their metabolites from the body. Toxicity assessment aims to evaluate the potential adverse effects of a drug candidate on various organs, tissues, or systems.

We can segment the early drug discovery pipeline into four main phases: target identification, hit discovery, hit-to-lead or lead identification, and lead optimization. Target identification designates a valid target whose activity is worth modulating to address some disease or disorder. Hit discovery uncovers chemical compounds with activity against the target. Lead identification selects the most promising hits and lead optimization improves their potency, selectivity, and ADMET properties to be suitable for preclinical study.

In virtual screening, we conducted our search across a chemical space consisting of an enormous set of molecules. In de novo design, we are still conducting an (informal) search, just not across the chemical space. We are now searching across the functional space of potential molecular properties. If our model “learns” which section of the functional space maps to molecules that have ideal binding affinity and safety, then perhaps it can reverse-engineer novel molecule structures in the chemical space that match our functional criteria.

Preclinical trials evaluate drug candidate safety and efficacy on model organisms. Phase I clinical trials evaluate drug candidate safety in its first exposure to humans. Phase II and Phase III clinical trials continue to collect data on safety while measuring drug candidate efficacy on larger groups of patients. The pass rate of our lead compounds decreases drastically as they progress beyond preclinical stages, along with an increase in the associated time to test them.

• Developing therapeutics entails a long, arduous process. Traditional development from ideation to market is costly (magnitude of billions of dollars), lengthy (10 to 15 years), and risky (attrition of over 90%). Through advances in AI, we can discover cures that have better safety profiles, address medical conditions or diseases with low coverage, and can reach patients quicker.
• Drug discovery can be thought of as a difficult search problem that exists at the intersection of the chemical search space of 10⁶³ medicinal compounds and the biological search space of 10⁵ targets.
• We can represent molecules computationally using SMILES, a text-based notation that encodes molecular structure as a character string, or as molecular graphs, where atoms are nodes and bonds are edges. These representations serve as input data for machine learning models.
• ML is a subfield of AI that enables computers to learn from and make decisions based on data, automatically and without explicit programming or rules on how to behave. Example ML algorithms include logistic regression and tree-based models. Deep learning is a subfield of ML that uses deep neural networks to extract complex patterns and representations from data.
• Applications of AI to drug design include molecule property prediction for virtual screening, creation of compound libraries with de novo molecule generation, synthesis pathway prediction, and protein folding simulation.
• We can segment the early drug discovery pipeline into four main phases: target identification, hit discovery, hit-to-lead or lead identification, and lead optimization. Target identification designates a valid target whose activity is worth modulating to address some disease or disorder. Hit discovery uncovers chemical compounds with activity against the target. Lead identification selects the most promising hits and lead optimization improves their potency, selectivity, and ADMET properties to be suitable for preclinical study.
• Popular, well-maintained chemical data repositories include ChEMBL, ChEBI, PubChem, Protein Data Bank (PDB), AlphaFoldDB, and ZINC. When using a new data source, learn how it was assembled and how quality is maintained. Garbage data in, garbage model out. See “Appendix B: Chemical Data Repositories” for more information.

FAQ