Overview

7 Deep transfer learning for NLP with the transformer and GPT

The chapter introduces transformer-based deep transfer learning as a major advance in NLP, emphasizing why these architectures became so influential despite having many more parameters than earlier convolutional and recurrent models. It frames the transformer as the key idea that replaced recurrence and convolution with attention, making large-scale NLP models easier to parallelize and more practical to train. The chapter also previews the main model families built on this foundation, focusing here on the transformer itself and GPT, while noting that BERT and multilingual BERT are covered separately.

At the architectural level, the transformer is explained as an encoder-decoder system built from stacked layers, input and output embeddings, and positional encodings that preserve word order without recurrence. The core mechanism is self-attention, where each token is compared against all others through query, key, and value vectors to capture contextual dependencies, such as linking an adjective to the noun it describes. Multihead attention extends this by allowing multiple relationship patterns to be learned at once, while residual connections and normalization help training stability. The chapter also shows how attention can be visualized with Python tools, making the model’s behavior more interpretable and concrete.

The practical sections apply pretrained transformer models to real tasks using the Hugging Face ecosystem. GPT is presented as a decoder-only transformer trained with a causal objective, making it especially effective for open-ended text generation, and the chapter demonstrates this through simple pipeline-based examples. It then extends GPT to dialogue with DialoGPT, showing how conversational history can be fed into the model to produce chatbot responses. A translation example using a pretrained encoder-decoder model illustrates how transformer checkpoints can be reused for low-resource language tasks, highlighting both the promise and the limitations of pretrained models when data is scarce or biased.

A high-level representation of the transformer architecture, showing stacked encoders, decoders, input/output embeddings, and positional encodings
07_01
A simplified decomposition of the encoder and decoder into self-attention, encoder-decoder attention, and feedforward neural networks
07_02
Self-attention visualization in the final encoding layer of the pretrained uncased BERT model for our example sentence. It reveals that “cells” is associated with “it” and “boring.” Note that this is a multihead view, with the shadings in each single column representing each head. Multiheaded attention is addressed in detail in section 7.1.2.
07_03
A visualization of the calculation of the self-attention weight of the word boring in our example sentence. Observe that the computations of these weights for different words can be carried out independently once key, value, and query vectors have been created. This is the root of the increased parallelizability of transformers over recurrent models. The attention coefficients are what is visualized as intensity of shading in any given column of the multihead attention in figure 7.3.
07_04
(Duplicated) Self-attention visualization in the final encoding layer of the pretrained uncased BERT model for our example sentence. It reveals that “cells” is associated with “it” and “boring.” Note that this is a multihead view, with the shadings in each single column representing each head.
07_04_07_03_duplicated
Vectorized self-attention calculation for the whole input sequence using matrices
07_05
A more detailed and accurate breakdown of each transformer encoder, now incorporating residual connections and normalization layers
07_06
(Duplicated, encoder-decoder attention highlighted) Simplified decomposition of the encoder and decoder into self-attention, encoder-decoder attention, and feedforward neural networks
07_06_007_02_duplicated
(Duplicated, slightly adjusted for encoder-decoder attention calculation) A visualization of the calculation of the encoder-decoder attention weight between the word boring in our example sentence and the output at position n. Observe that the computations of these weights for different words can be carried out independently once key, value, and query vectors have been created. This is the root of the increased parallelizability of transformers over recurrent models.
07_06_007_04_duplicated
(Duplicated, masked layer highlighted) The simplified decomposition of the encoder and decoder into self-attention, encoder-decoder attention, and feedforward neural networks
07_06_07_02_duplicated2
(Duplicated again, modified for masked self-attention) A masked self-attention visualization for our example sentence, showing future tokens being masked for causality.
07_06_07_03_duplicated2
A high-level representation of the GPT architecture, showing stacked decoders, input embeddings, and positional encodings. The output from the top can be used for both text prediction/generation and classification.
07_07

Summary

  • The transformer architecture uses self-attention to build bidirectional context for understanding text. This has enabled it to become a dominant language model in NLP recently.
  • The transformer allows tokens in a sequence to be processed independently of each other. This achieves greater parallelizability than bi-LSTMs, which process tokens in order.
  • The transformer is a good choice for translation applications.
  • The Generative Pretrained Transformer uses the causal modeling objective during training. This makes it the model of choice for text generation, such as chatbot applications.

1. A. Vaswani et al., “Attention Is All You Need,” NeurIPS (2017).

2. A. Radford et al., “Improving Language Understanding by Generative Pre-Training,” arXiv (2018).

3. M. E. Peters et al., “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding,” Proc. of NAACL-HLT (2019).

5. A. Radfordet al., “Improving Language Understanding by Generative Pre-Training,” arXiv (2018).

6. A. Vaswaniet al., “Attention Is All You Need,” NeurIPS (2017).

11. A. Radford et al., “Improving Language Understanding by Generative Pre-Training,” arXiv (2018).

12. Y. Zhang et al., “DialoGPT: Large-Scale Generative Pretraining for Conversational Response Generation,” arXiv (2019).

FAQ

What is the transformer architecture and why is it important for NLP?The transformer is a neural network architecture that replaces recurrent and convolutional components with attention mechanisms. It is important because it scales well on parallel computing hardware, handles long-range dependencies effectively, and has become a foundation for many state-of-the-art NLP models.
How does the transformer differ from traditional recurrent neural networks?Unlike recurrent models, the transformer does not process tokens strictly one at a time through recurrence. Instead, it uses attention to relate all tokens in a sequence to one another, which makes training and inference more parallelizable and often more efficient.
What is self-attention in the transformer?Self-attention is a mechanism where each token attends to other tokens in the same sequence to learn contextual relationships. It helps the model determine which words are most relevant to each other when building representations of the input.
Why are positional encodings needed in the transformer?Because self-attention alone does not preserve word order, positional encodings are added to token embeddings to provide information about the position of each token in the sequence. This lets the model retain sequential awareness without recurrence.
What is multihead attention and why is it useful?Multihead attention runs several attention operations in parallel, allowing the model to capture different types of relationships at once, such as subject-verb or noun-adjective links. This gives the transformer richer representations than single-head attention.
What do residual connections and normalization do in transformer layers?Residual connections help gradients flow through deep networks more easily, which reduces vanishing or exploding gradient problems. Normalization keeps feature scales more stable across layers, improving training behavior and model robustness.
How is GPT different from BERT in terms of training objective?GPT is trained with a causal language modeling objective, predicting the next token in a sequence. BERT, by contrast, uses masked language modeling and next sentence prediction, which makes BERT better for understanding tasks and GPT better suited for generation.
Why is GPT especially good for text generation?GPT is designed to predict the next token based on previous tokens only, which naturally matches the process of generating text from a prompt. Its decoder-only, causal structure makes it effective for open-ended generation tasks.
How can pretrained GPT be used in the transformers library for text generation?You can load GPT through the transformers pipeline API with a text-generation task and provide a prompt. The model then generates continuation text, and you can control the output length with parameters like max_length.
What is DialoGPT and how is it used for chatbots?DialoGPT is a GPT-based model fine-tuned on conversational data for multi-turn dialogue. It can be used as a chatbot by feeding the conversation history into the model and generating the next response turn by turn.

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Transfer Learning for Natural Language Processing Transfer Learning for Natural Language Processing ebook for free
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Transfer Learning for Natural Language Processing Transfer Learning for Natural Language Processing ebook for free
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Transfer Learning for Natural Language Processing Transfer Learning for Natural Language Processing ebook for free