Natural language processing sits within the broader history of artificial intelligence, alongside machine learning and deep learning, and the chapter begins by clarifying how these areas relate. AI originally relied on hand-coded symbolic rules, but those systems struggled with perception-heavy problems like vision, speech, and language. Machine learning shifted the field toward learning from examples, and deep learning later expanded what could be handled by automatically learning layered representations from large datasets and powerful hardware. In this context, NLP is presented as the branch of AI focused on helping computers understand and work with human language, powering tasks such as translation, speech recognition, classification, question answering, summarization, and dependency parsing.
The chapter then traces how NLP evolved from rule-based systems to feature engineering and finally to neural methods that reduced the need for manual design. Early approaches relied on labeled data and shallow models, while later techniques such as word embeddings, sequence-to-sequence models, attention, and transformers improved performance by learning richer representations from unlabeled or weakly labeled data. A key theme is that transfer learning emerged as a practical response to the cost of training large models from scratch. By first learning general language knowledge on massive corpora and then adapting that knowledge to specific downstream tasks, practitioners could achieve strong results with far fewer labels and far fewer resources than before.
Transfer learning is described as reusing knowledge from a source setting to improve performance in a target setting, where a setting combines task, domain, and language. The chapter emphasizes that this idea has long been common in computer vision, especially through pretrained ImageNet models, which are often reused via feature extraction or fine-tuning rather than trained anew. That computer vision experience helped shape modern NLP transfer learning methods and heuristics, such as deciding which layers to freeze or unfreeze based on data size and similarity between source and target problems. The chapter closes by explaining why the topic matters now: pretrained language models and fine-tuning methods have made high-quality NLP more accessible, while new language phenomena, social media, misinformation, and broader deployment pressures continue to make transfer learning increasingly important.
An illustration of the advantages of the transfer learning paradigm—shown in the bottom panel—where information is shared between systems trained for different tasks/domains, versus the traditional paradigm—shown in the top panel—where training occurs in parallel between tasks/domains. In the transfer learning paradigm, reduction in data and computing requirements can be achieved via the information/knowledge sharing. For instance, we expect a person to learn to play the drums more easily if they know how to play the piano first.
A Venn diagram visualization of the terms natural language processing (NLP), artificial intelligence (AI), machine learning, and deep learning relative to each other. Symbolic AI is also shown.
An illustrative example of a major motivating task in machine learning: finding a decision boundary in the hypothesis set to effectively separate different types of points from each other. In the case shown in this figure, the hypothesis set may be the set of arcs.
Example decision tree structure that classifies the input {A,B} in category 1 if A<10, category 2 if A>=10 while B<25, and category 3 otherwise
Visualization of a simple feedforward neural network with two fully connected hidden layers (left). On the right is a summarized equivalent representation, which we will often employ to simplify diagrams.
Visualization of the various transfer learning heuristics applicable in computer vision for feedforward neural network architectures, which we will draw on in NLP whenever possible. A threshold is moved to the left, with more availability of training data in the target domain, and all parameters to the right of it are retrained, with the exception of those that are discarded due to increasing dissimilarity between source and target domains.
Summary
- Artificial intelligence (AI) holds the promise of fundamentally transforming our society. To democratize the benefits of this transformation, we must ensure that state-of-the-art advances are accessible to everyone, regardless of language, access to massive computing resources, and country of origin.
- Machine learning is the dominant modern paradigm in AI, which, rather than explicitly programming a computer for every possible scenario, trains it to associate input to output signals by seeing many examples of such corresponding input-output pairs.
- Natural language processing (NLP), the subfield of AI we will be discussing in this book, deals with the analysis and processing of human natural language data and is one of the most active areas of AI research today.
- A recently popularized paradigm in NLP, transfer learning, enables you to adapt or transfer the knowledge acquired from one set of tasks or domains to a different set of tasks or domains. This is a big step forward for the democratization of NLP and, more widely, AI, allowing knowledge to be reused in new settings at a fraction of the previously required resources, which may not be available to everyone.
- Key modeling frameworks enabling transfer learning in NLP include ELMo and BERT.
- The recent rise in the importance of social media has changed the definition of what is considered natural language. Now, billions of people express themselves online using emoticons, newly invented slang, and deliberately misspelled words. All these present new challenges, which we must take into account when developing new transfer learning techniques for NLP.
- Transfer learning is relatively well understood in computer vision, and whenever possible, we should draw on this body of knowledge when experimenting with new transfer techniques for NLP.
2. J. Devlin et al., “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding,” arXiv (2018).
5. M. Pagliardini et al., “Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features,” Proc. of NAACL-HLT (2018).
7. I. Sutskever et al., “Sequence to Sequence Learning with Neural Networks,” NeurIPS Proceedings (2014).
9. X. Zhang et al., “Character-Level Convolutional Networks for Text Classification,” NeurIPS Proceedings (2015).
10. P. Azunre et al., “Semantic Classification of Tabular Datasets via Character-Level Convolutional Neural Networks,” arXiv (2019).
12. J. Howard et al., “Universal Language Model Fine-Tuning for Text Classification,” Proc. of the 56th Annual Meeting of the Association for Computational Linguistics (2018).
13. J. Devlin et al., “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding,” Proc. of NAACL-HLT (2019).
14. J. Deng et al., “ImageNet: A Large-Scale Hierarchical Image Database,” Proc. of NAACL-HLT (2019).
15. K. Schaffer, Data versus Democracy:- How Big Data Algorithms Shape Opinions and Alter the Course of History (New York: Apress, 2019).
Transfer Learning for Natural Language Processing ebook for free