Build a Large Language Model (From Scratch) you own this product

Sebastian Raschka
  • September 2024
  • ISBN 9781633437166
  • 368 pages
  • printed in black & white

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


Look inside
Learn how to create, train, and tweak large language models (LLMs) by building one from the ground up!

In Build a Large Language Model (from Scratch) bestselling author Sebastian Raschka guides you step by step through creating your own LLM. Each stage is explained with clear text, diagrams, and examples. You’ll go from the initial design and creation, to pretraining on a general corpus, and on to fine-tuning for specific tasks.

Build a Large Language Model (from Scratch) teaches you how to:

  • Plan and code all the parts of an LLM
  • Prepare a dataset suitable for LLM training
  • Fine-tune LLMs for text classification and with your own data
  • Use human feedback to ensure your LLM follows instructions
  • Load pretrained weights into an LLM

Build a Large Language Model (from Scratch) takes you inside the AI black box to tinker with the internal systems that power generative AI. As you work through each key stage of LLM creation, you’ll develop an in-depth understanding of how LLMs work, their limitations, and their customization methods. Your LLM can be developed on an ordinary laptop, and used as your own personal assistant.

about the technology

Physicist Richard P. Feynman reportedly said, “I don’t understand anything I can’t build.” Based on this same powerful principle, bestselling author Sebastian Raschka guides you step by step as you build a GPT-style LLM that you can run on your laptop. This is an engaging book that covers each stage of the process, from planning and coding to training and fine-tuning.

about the book

Build a Large Language Model (From Scratch) is a practical and eminently-satisfying hands-on journey into the foundations of generative AI. Without relying on any existing LLM libraries, you’ll code a base model, evolve it into a text classifier, and ultimately create a chatbot that can follow your conversational instructions. And you’ll really understand it because you built it yourself!

For deeper understanding and better learning we provide a built-in testing system into liveBook, the online version of this book. Separately, you can download a free PDF Test Yourself guide on this book from here.

Frequently Asked Questions

Key challenges include addressing biases, ensuring safety and ethical use, maintaining transparency and explainability, and ensuring data privacy and security.

Yes, after finishing this bestselling LLM book by Sebastian Raschka you should have a solid understanding of how to create a large language model, which is a fundamental component of building a chatbot. The book covers the necessary concepts and techniques to develop and train your own language model, which can then be used as the basis for a chatbot.

LLMs learn from massive datasets of text and code, identifying patterns and relationships between words and phrases. They then use this knowledge to generate new text, translate languages, write different kinds of creative content, and answer your questions in an informative way.

Using this book as a guide, you can build a sophisticated chatbot capable of understanding and generating human-like text. This includes tasks such as machine translation, text summarization, sentiment analysis, and content creation. The chatbot leverages the advanced capabilities of large language models to perform these tasks effectively.

LLMs have the potential to revolutionize various fields, including customer service, education, healthcare, and research. They can be used for tasks such as chatbots, language translation, content creation, and even drug discovery.

Yes, with the help from this book, you can build a chatbot similar to GPT-2 in terms of capabilities. GPT-3 is significantly larger and more complex, requiring substantial computational resources and data, which is beyond the scope of this book. However, the foundational knowledge you gain will be applicable to understanding and working with models like GPT-3.

Ethical concerns include the potential for job displacement, the spread of misinformation, the perpetuation of biases, and the misuse of LLMs for malicious purposes.

Yes, using a GPU is highly recommended for training large language models as discussed in the book. GPUs significantly speed up the training process due to their ability to handle parallel computations efficiently, which is crucial for deep learning tasks.

The future of LLM research and development is likely to focus on improving their capabilities, addressing their limitations, and ensuring their responsible and ethical use. This may involve developing more advanced models, exploring new architectures, and developing robust safety and governance mechanisms.

This book teaches you how to build a model from the ground up, rather than just fine-tuning an existing model. It covers the fundamental concepts and techniques needed to develop and train your own large language model from scratch.

what's inside

  • Plan and code an LLM comparable to GPT-2
  • Load pretrained weights
  • Construct a complete training pipeline
  • Fine-tune your LLM for text classification
  • Develop LLMs that follow human instructions

about the reader

Readers need intermediate Python skills and some knowledge of machine learning. The LLM you create will run on any modern laptop and can optionally utilize GPUs.

about the author

Sebastian Raschka is a Staff Research Engineer at Lightning AI, where he works on LLM research and develops open-source software.

The technical editor on this book was David Caswell.

Truly inspirational! It motivates you to put your new skills into action.

Benjamin Muskalla, Senior Engineer, GitHub

The most understandable and comprehensive explanation of language models yet! Its unique and practical teaching style achieves a level of understanding you can’t get any other way.

Cameron Wolfe, Senior Scientist, Netflix

Sebastian combines deep knowledge with practical engineering skills and a knack for making complex ideas simple. This is the guide you need!

Chip Huyen, author of Designing Machine Learning Systems and AI Engineering

Definitive, up-to-date coverage. Highly recommended!

Dr. Vahid Mirjalili, Senior Data Scientist, FM Global

choose your plan

team

monthly
annual
$49.99
$399.99
only $33.33 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • Build a Large Language Model (From Scratch) ebook for free

choose your plan

team

monthly
annual
$49.99
$399.99
only $33.33 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • Build a Large Language Model (From Scratch) ebook for free

choose your plan

team

monthly
annual
$49.99
$399.99
only $33.33 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • Build a Large Language Model (From Scratch) ebook for free