1 Understanding Large Language Models

Large language models are deep neural networks trained on vast text corpora to produce coherent, context-aware language. They mark a shift from earlier, task-specific NLP systems by generalizing across translation, summarization, question answering, sentiment analysis, and more. Their apparent “understanding” reflects statistical pattern mastery rather than human comprehension, and their gains stem from scaling data and parameters alongside advances in deep learning, especially the transformer architecture.

The transformer enables models to capture long-range dependencies via self-attention. Its original encoder–decoder design inspired two influential families: BERT-like models that excel at understanding and classification through masked word prediction, and GPT-like models that generate text autoregressively via next-word prediction. Despite this seemingly simple pretraining objective, GPT models exhibit zero-shot and few-shot abilities and show emergent behaviors such as translation, enabled by large-scale training, tokenization, and diverse datasets.

Building and deploying LLMs typically follows a two-stage process: pretraining on raw text to create a general foundation model, followed by finetuning on labeled data for instruction following or classification. Custom models can outperform general ones in domain tasks and offer benefits in privacy, on-device deployment, latency, cost control, and autonomy. While pretraining is resource-intensive, many open-source models and pretrained weights make practical development accessible. This book implements the end-to-end pipeline step by step: data preparation and attention, educational-scale pretraining and evaluation, and finetuning a foundation model into an assistant or classifier.

As this hierarchical depiction of the relationship between the different fields suggests, LLMs represent a specific application of deep learning techniques, leveraging their ability to process and generate human-like text. Deep learning is a specialized branch of machine learning that focuses on using multi-layer neural networks. And machine learning and deep learning are fields aimed at implementing algorithms that enable computers to learn from data and perform tasks that typically require human intelligence.

LLM interfaces enable natural language communication between users and AI systems. This screenshot shows ChatGPT writing a poem according to a user's specifications.

Pretraining an LLM involves next-word prediction on large text datasets. A pretrained LLM can then be finetuned using a smaller labeled dataset.

A simplified depiction of the original transformer architecture, which is a deep learning model for language translation. The transformer consists of two parts, an encoder that processes the input text and produces an embedding representation (a numerical representation that captures many different factors in different dimensions) of the text that the decoder can use to generate the translated text one word at a time. Note that this figure shows the final stage of the translation process where the decoder has to generate only the final word ("Beispiel"), given the original input text ("This is an example") and a partially translated sentence ("Das ist ein"), to complete the translation.

A visual representation of the transformer's encoder and decoder submodules. On the left, the encoder segment exemplifies BERT-like LLMs, which focus on masked word prediction and are primarily used for tasks like text classification. On the right, the decoder segment showcases GPT-like LLMs, designed for generative tasks and producing coherent text sequences.

In addition to text completion, GPT-like LLMs can solve various tasks based on their inputs without needing retraining, finetuning, or task-specific model architecture changes. Sometimes, it is helpful to provide examples of the target within the input, which is known as a few-shot setting. However, GPT-like LLMs are also capable of carrying out tasks without a specific example, which is called zero-shot setting.

In the next-word pretraining task for GPT models, the system learns to predict the upcoming word in a sentence by looking at the words that have come before it. This approach helps the model understand how words and phrases typically fit together in language, forming a foundation that can be applied to various other tasks.

The GPT architecture employs only the decoder portion of the original transformer. It is designed for unidirectional, left-to-right processing, making it well-suited for text generation and next-word prediction tasks to generate text in iterative fashion one word at a time.

The stages of building LLMs covered in this book include implementing the LLM architecture and data preparation process, pretraining an LLM to create a foundation model, and finetuning the foundation model to become a personal assistant or text classifier.

• LLMs have transformed the field of natural language processing, which previously mostly relied on explicit rule-based systems and simpler statistical methods. The advent of LLMs introduced new deep learning-driven approaches that led to advancements in understanding, generating, and translating human language.
• Modern LLMs are trained in two main steps.
• First, they are pretrained on a large corpus of unlabeled text by using the prediction of the next word in a sentence as a "label."
• Then, they are finetuned on a smaller, labeled target dataset to follow instructions or perform classification tasks.
• LLMs are based on the transformer architecture. The key idea of the transformer architecture is an attention mechanism that gives the LLM selective access to the whole input sequence when generating the output one word at a time.
• The original transformer architecture consists of an encoder for parsing text and a decoder for generating text.
• LLMs for generating text and following instructions, such as GPT-3 and ChatGPT, only implement decoder modules, simplifying the architecture.
• Large datasets consisting of billions of words are essential for pretraining LLMs. In this book, we will implement and train LLMs on small datasets for educational purposes but also see how we can load openly available model weights.
• While the general pretraining task for GPT-like models is to predict the next word in a sentence, these LLMs exhibit "emergent" properties such as capabilities to classify, translate, or summarize texts.
• Once an LLM is pretrained, the resulting foundation model can be finetuned more efficiently for various downstream tasks.
• LLMs finetuned on custom datasets can outperform general LLMs on specific tasks.

FAQ