8 Generating Protein Structures
This chapter broadens the discussion of domain-specific small language models from code generation to chemistry, drug discovery, protein engineering, antibody design, and materials science. It emphasizes that many scientific problems can be reformulated as sequence-generation tasks, allowing Transformer-based models to operate on specialized textual representations such as amino acid sequences, antibody sequences, SMILES-like chemical encodings, and crystallographic information files. A central message is that smaller open models, when trained or fine-tuned on the right domain data, can outperform or be more practical than large closed commercial models for highly specialized scientific workflows.
The chapter first demonstrates protein generation with ProtGPT2, a GPT-2-style decoder-only model trained on raw protein sequences to generate novel proteins in a zero-shot manner. It shows that inference can be performed much like ordinary text generation, using prompts, sampling parameters, and sequence outputs, while quality can be roughly inspected with metrics such as perplexity. It then explores AntibodyGPT, based on ProGen2 and fine-tuned for therapeutic antibody sequence generation. This example highlights the importance of GPU acceleration, domain-specific validation tools such as antibody numbering and classification systems, and practical inference optimization techniques, including CUDA allocator configuration and 8-bit quantization, which substantially reduce model size and latency.
The final major example focuses on CrystaLLM, a decoder-only Transformer trained to generate crystal structures from textual representations of crystallographic information files. The chapter explains how prompts based on cell composition can be used to generate raw CIF files, which then require post-processing and validation to ensure structural consistency and physical plausibility. It also discusses improved sampling through Monte Carlo Tree Search and shows how pretrained checkpoints can be adapted for use with common Transformer tooling. Overall, the chapter reinforces that domain-specific SLMs can support real-world scientific generation tasks while remaining practical to run locally, optimize, validate, and integrate into broader engineering workflows.
The different options available for the ProGen2 and the Antibody Generator model families.
The output of the online ANARCI tool for one of the antibody sequences generated in our example.
The CrystaLLM training (a) and generation with MCTS decoding (b) processes.
Summary
- Chemistry language models adapt NLP techniques to represent molecular structures as text sequences for drug discovery tasks.
- ProtGPT2 generates protein sequences using a decoder-only GPT-2 architecture trained on 87162 protein sequences from UniRef50 database.
- Protein generation requires prompting with endoftext tokens and produces amino acid sequences that follow natural protein patterns.
- Perplexity scoring evaluates generated protein sequences with lower scores indicating better quality outputs.
- AntibodyGPT fine-tunes ProGen2 models on 5000 experimentally resolved crystal structures for therapeutic antibody generation.
- ProGenForCausalLM class enables loading and inference with ProGen2-based models from custom repositories.
- CUDA memory allocation settings can improve inference latency by enabling expandable segments for better memory management.
- 8-bit quantization reduces AntibodyGPT model size from 617MB to 156MB with 40% improvement in average latency.
- ANARCI tool validates generated antibody sequences for conformity to known antibody structural patterns.
- CrystaLLM generates crystal structures from textual representations of crystallographic information files.
- CIF file generation involves tokenizing crystallographic data and using autoregressive prediction for structure completion.
- Monte Carlo Tree Search algorithm improves structural sampling quality for crystal structure generation.
- Chemical structure validation requires domain-specific tools rather than standard text generation metrics.
- Domain-specific models achieve effective results with smaller parameter counts than generalist language models.
- Specialized chemistry models require subject matter expert validation for biological relevance and functional viability.
FAQ
What is the main focus of Chapter 8, “Generating Protein Structures”?
The chapter shows how domain-specific small language models (SLMs) can be applied to chemistry, drug R&D, protein generation, antibody generation, and crystal structure generation. It demonstrates models such as ProtGPT2, AntibodyGPT, and CrystaLLM, and discusses practical inference considerations such as GPU usage, latency, CUDA memory allocation, validation tools, and quantization.Why are small open-source language models useful in chemistry and drug discovery?
Small open-source models are useful because chemistry and drug discovery often require specialized data, task-specific tuning, and domain-specific representations that large commercial models may not handle well. Open models also give researchers more control over training, fine-tuning, deployment, and validation, while avoiding the high costs and closed nature of commercial LLMs.What is ProtGPT2 and what is it used for?
ProtGPT2 is a decoder-only model based on the GPT-2 architecture with 738 million parameters. It is trained to “speak” the protein language and can generate de novo protein sequences. The model was pretrained in a self-supervised way on raw protein sequences from the UniRef50 database and can generate protein sequences in a zero-shot fashion.How can ProtGPT2 be used to generate protein sequences?
ProtGPT2 can be loaded from the Hugging Face Hub using the Transformers text-generation pipeline. A prompt such as<|endoftext|> can be passed to the model with generation parameters like max_length, top_k, repetition_penalty, and num_return_sequences. The process is similar to generating text or code, except the output is a sequence of amino acids.How is the quality of ProtGPT2-generated protein sequences evaluated in the chapter?
The chapter uses perplexity as a basic metric for evaluating generated protein sequences. As with text generation, lower perplexity is considered better. However, for highly specialized biological tasks, perplexity alone is not always sufficient, and additional domain-specific validation may be required.What is AntibodyGPT and how was it created?
AntibodyGPT, also called the Antibody Generator, is a specialized protein generation model for creating therapeutic antibody sequences. It is based on Salesforce’s ProGen2 models, which are decoder-only protein language models. AntibodyGPT versions were created by fine-tuning ProGen2 small, medium, and large models on 5,000 experimentally resolved crystal structures of antibodies and antigens.Why does AntibodyGPT require more inference optimization than ProtGPT2?
Although the small AntibodyGPT model is smaller than ProtGPT2, generating antibody sequences can still be slow without hardware acceleration. In the chapter’s example, generating just two target sequences can take hours on a Colab CPU runtime, so GPU acceleration is recommended. The chapter explores CUDA allocator settings and 8-bit quantization to reduce latency.How can AntibodyGPT inference latency be improved?
The chapter discusses two improvements. First, settingPYTORCH_CUDA_ALLOC_CONF=expandable_segments:True can reduce CUDA memory fragmentation and improve latency. Second, using 8-bit quantization with the BitsAndBytes library significantly reduces model size and can lower average latency by about 40% compared with the non-quantized model after enabling expandable CUDA segments.
Domain-Specific Small Language Models ebook for free
