Overview

1 What is transfer learning?

Natural language processing sits within the broader history of artificial intelligence, alongside machine learning and deep learning, and the chapter begins by clarifying how these areas relate. AI originally relied on hand-coded symbolic rules, but those systems struggled with perception-heavy problems like vision, speech, and language. Machine learning shifted the field toward learning from examples, and deep learning later expanded what could be handled by automatically learning layered representations from large datasets and powerful hardware. In this context, NLP is presented as the branch of AI focused on helping computers understand and work with human language, powering tasks such as translation, speech recognition, classification, question answering, summarization, and dependency parsing.

The chapter then traces how NLP evolved from rule-based systems to feature engineering and finally to neural methods that reduced the need for manual design. Early approaches relied on labeled data and shallow models, while later techniques such as word embeddings, sequence-to-sequence models, attention, and transformers improved performance by learning richer representations from unlabeled or weakly labeled data. A key theme is that transfer learning emerged as a practical response to the cost of training large models from scratch. By first learning general language knowledge on massive corpora and then adapting that knowledge to specific downstream tasks, practitioners could achieve strong results with far fewer labels and far fewer resources than before.

Transfer learning is described as reusing knowledge from a source setting to improve performance in a target setting, where a setting combines task, domain, and language. The chapter emphasizes that this idea has long been common in computer vision, especially through pretrained ImageNet models, which are often reused via feature extraction or fine-tuning rather than trained anew. That computer vision experience helped shape modern NLP transfer learning methods and heuristics, such as deciding which layers to freeze or unfreeze based on data size and similarity between source and target problems. The chapter closes by explaining why the topic matters now: pretrained language models and fine-tuning methods have made high-quality NLP more accessible, while new language phenomena, social media, misinformation, and broader deployment pressures continue to make transfer learning increasingly important.

An illustration of the advantages of the transfer learning paradigm—shown in the bottom panel—where information is shared between systems trained for different tasks/domains, versus the traditional paradigm—shown in the top panel—where training occurs in parallel between tasks/domains. In the transfer learning paradigm, reduction in data and computing requirements can be achieved via the information/knowledge sharing. For instance, we expect a person to learn to play the drums more easily if they know how to play the piano first.
01_01
A Venn diagram visualization of the terms natural language processing (NLP), artificial intelligence (AI), machine learning, and deep learning relative to each other. Symbolic AI is also shown.
01_02
An illustrative example of a major motivating task in machine learning: finding a decision boundary in the hypothesis set to effectively separate different types of points from each other. In the case shown in this figure, the hypothesis set may be the set of arcs.
01_03
Example decision tree structure that classifies the input {A,B} in category 1 if A<10, category 2 if A>=10 while B<25, and category 3 otherwise
01_04
Visualization of a simple feedforward neural network with two fully connected hidden layers (left). On the right is a summarized equivalent representation, which we will often employ to simplify diagrams.
01_05
Visualization of the various transfer learning heuristics applicable in computer vision for feedforward neural network architectures, which we will draw on in NLP whenever possible. A threshold is moved to the left, with more availability of training data in the target domain, and all parameters to the right of it are retrained, with the exception of those that are discarded due to increasing dissimilarity between source and target domains.
01_06

Summary

  • Artificial intelligence (AI) holds the promise of fundamentally transforming our society. To democratize the benefits of this transformation, we must ensure that state-of-the-art advances are accessible to everyone, regardless of language, access to massive computing resources, and country of origin.
  • Machine learning is the dominant modern paradigm in AI, which, rather than explicitly programming a computer for every possible scenario, trains it to associate input to output signals by seeing many examples of such corresponding input-output pairs.
  • Natural language processing (NLP), the subfield of AI we will be discussing in this book, deals with the analysis and processing of human natural language data and is one of the most active areas of AI research today.
  • A recently popularized paradigm in NLP, transfer learning, enables you to adapt or transfer the knowledge acquired from one set of tasks or domains to a different set of tasks or domains. This is a big step forward for the democratization of NLP and, more widely, AI, allowing knowledge to be reused in new settings at a fraction of the previously required resources, which may not be available to everyone.
  • Key modeling frameworks enabling transfer learning in NLP include ELMo and BERT.
  • The recent rise in the importance of social media has changed the definition of what is considered natural language. Now, billions of people express themselves online using emoticons, newly invented slang, and deliberately misspelled words. All these present new challenges, which we must take into account when developing new transfer learning techniques for NLP.
  • Transfer learning is relatively well understood in computer vision, and whenever possible, we should draw on this body of knowledge when experimenting with new transfer techniques for NLP.

1. K. Schwab, The Fourth Industrial Revolution (Geneva: World Economic Forum, 2016).

2. J. Devlin et al., “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding,” arXiv (2018).

3. F. Chollet, Deep Learning with Python (New York: Manning Publications, 2018).

4. T. Mikolov et al., “Efficient Estimation of Word Representations in Vector Space,” arXiv (2013).

5. M. Pagliardini et al., “Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features,” Proc. of NAACL-HLT (2018).

6. Q. V. Le et al., “Distributed Representations of Sentences and Documents,” arXiv (2014).

7. I. Sutskever et al., “Sequence to Sequence Learning with Neural Networks,” NeurIPS Proceedings (2014).

8. A. Vaswani et al., “Attention Is All You Need,” NeurIPS Proceedings (2017).

9. X. Zhang et al., “Character-Level Convolutional Networks for Text Classification,” NeurIPS Proceedings (2015).

10. P. Azunre et al., “Semantic Classification of Tabular Datasets via Character-Level Convolutional Neural Networks,” arXiv (2019).

11. M. E. Peters et al., “Deep Contextualized Word Representations,” Proc. of NAACL-HLT (2018).

12. J. Howard et al., “Universal Language Model Fine-Tuning for Text Classification,” Proc. of the 56th Annual Meeting of the Association for Computational Linguistics (2018).

13. J. Devlin et al., “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding,” Proc. of NAACL-HLT (2019).

14. J. Deng et al., “ImageNet: A Large-Scale Hierarchical Image Database,” Proc. of NAACL-HLT (2019).

15. K. Schaffer, Data versus Democracy:- How Big Data Algorithms Shape Opinions and Alter the Course of History (New York: Apress, 2019).

FAQ

```html
What is transfer learning in the context of NLP?Transfer learning in NLP is the practice of taking knowledge learned from one setting—such as a task, domain, or language—and reusing it to improve performance on a different NLP problem. It helps a model solve a target task more efficiently than training from scratch.
Why is transfer learning important for natural language processing?It reduces the need for large amounts of labeled data, computing power, time, and cost. This makes it possible for smaller teams and individual practitioners to achieve strong results without training massive models from scratch.
How does transfer learning relate to the idea of source and target settings?The source setting is the original task, domain, or language where knowledge is learned. The target setting is the new problem where that knowledge is reused. Transfer is easier when the source and target settings are similar.
What kinds of NLP tasks are commonly discussed in this chapter?The chapter highlights tasks such as part-of-speech tagging, named entity recognition, sentiment analysis, text classification, summarization, machine translation, question answering, chatbot systems, speech recognition, language modeling, and dependency parsing.
How is NLP positioned within the broader field of artificial intelligence?NLP is a subfield of AI focused on helping computers understand and process human language. It overlaps with machine learning and deep learning, and it has benefited greatly from both rule-based methods and data-driven approaches.
What role did deep learning play in the evolution of NLP?Deep learning transformed NLP by allowing models to learn useful representations automatically from data. This reduced the dependence on manual feature engineering and led to major improvements in difficult tasks such as machine translation and text classification.
How did earlier NLP methods differ from modern transfer learning approaches?Earlier methods often relied on handcrafted rules or manually engineered features, which were task-specific and hard to scale. Modern transfer learning reuses pretrained representations and fine-tunes them for new tasks, making learning more efficient and flexible.
What are some examples of pretrained NLP models mentioned in the chapter?The chapter mentions word2vec, sent2vec, doc2vec, ELMo, ULMFiT, GPT, and BERT. These models learn useful language representations from large datasets and can then be adapted to downstream tasks.
Why was 2018 a turning point for NLP transfer learning?In 2018, pretrained language models became widely available along with practical fine-tuning methods. This made state-of-the-art NLP techniques accessible to many more practitioners and sparked major progress in the field.
How does transfer learning in computer vision help explain transfer learning in NLP?Computer vision provided an earlier and successful model for transfer learning, where pretrained models like ImageNet networks are reused for new tasks. NLP borrowed this idea by using pretrained language models and fine-tuning them for specific language tasks.
```

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Transfer Learning for Natural Language Processing Transfer Learning for Natural Language Processing ebook for free
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Transfer Learning for Natural Language Processing Transfer Learning for Natural Language Processing ebook for free
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Transfer Learning for Natural Language Processing Transfer Learning for Natural Language Processing ebook for free