Overview

4 Shallow transfer learning for NLP

This chapter introduces shallow transfer learning as a practical way to reuse pretrained knowledge in relatively simple neural and non-neural pipelines. It frames transfer learning in three main settings: cross-lingual learning, multitask learning, and domain adaptation, then focuses on the latter two while using familiar NLP classification problems as running examples. The overall goal is to show how pretrained representations and shared structure can improve performance without requiring very deep architectures.

First, the chapter explores semisupervised learning with pretrained embeddings. It revisits IMDB sentiment classification and shows how pretrained word embeddings can replace bag-of-words features, with words mapped to vectors and then aggregated into review-level features. It compares common embedding families such as word2vec, GloVe, and fastText, and then moves to higher-level representations like sent2vec and doc2vec-style ideas, where entire sentences or documents are embedded directly. Using these richer embeddings and simple downstream classifiers generally improves results over naive averaging of word vectors and can outperform a basic bag-of-words baseline.

The chapter then turns to multitask learning and domain adaptation. In multitask learning, a single shallow neural model is trained on multiple tasks at once using hard parameter sharing, so shared layers learn more general representations while task-specific heads handle each output. Although this can slightly reduce performance on individual tasks, it can yield features that transfer better to new problems. Finally, the chapter demonstrates domain adaptation by transferring an IMDB-trained sentiment model to book reviews from the Multi-Domain Sentiment Dataset. A direct zero-shot transfer already works moderately well, and an autoencoder-based projection into the source feature space yields a small but consistent improvement, illustrating how source-domain knowledge can help when target-domain labeled data is scarce or unavailable.

Visualizing the categorization of transfer learning into multitask learning, domain adaptation, and cross-lingual learning
04_01
Sequence of typical steps for semisupervised learning with word, sentence, or document embeddings
04_02
The general form of neural multitask learning we will employ—hard parameter sharing (in this case, three tasks)
04_03
The specific form of neural multitask hard parameter sharing we will employ, with two specific tasks shown—IMDB reviews and email spam classification
04_04
A visualization of the process of one-hot encoding characters into row-vector representations. The process replaces every character in the vocabulary with a sparse vector with a dimension equal to the size of the vocabulary. 1 is placed in the column corresponding to the vocabulary character index.
04_05

Summary

  • Pretrained word embeddings, as well as embeddings at higher levels of text—such as sentences—are ubiquitous in NLP and can be used to convert text into numbers/vectors. This simplifies further processing to extract meaning from them.
  • Such an extraction represents a form of semisupervised shallow transfer learning, which has been widely used in practice with immense success.
  • Techniques like hard-parameter sharing and soft-parameter sharing allow us to create multitask learning systems, which have potential benefits, including simplified engineering design, improved generalization, and reduced overfitting.
  • Sometimes it may be possible to achieve zero-shot transfer learning with no labeled data in the target domain, which is an ideal scenario because labeled data can be expensive to collect.
  • It is sometimes possible to modify or adapt data in a target domain to be more similar to the data in the source domain, for instance, via projection methods such as autoencoders, and this can improve performance.

1. S. J. Pan and Q. Yang, “A Survey on Transfer Learning,” IEEE Transactions on Knowledge and Data Engineering (2009).

2. S. Ruder, “Neural Transfer Learning for Natural Language Processing,” National University of Ireland, Galway, Ireland (2019).

3. D. Wang and T. F. Zheng, “Transfer Learning for Speech and Language Processing,” Proceedings of 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA).

4. Jing Wang, Haibo Hea Danil, and V.Prokhorov, “A Folded Neural Network Autoencoder for Dimensionality Reduction,” Procedia Computer Science 13 (2012): 120-27.

5. Z. Yin and Y. Shen, “On the Dimensionality of Word Embedding,” 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montreal, Canada.

7. J. Alabi et al., “Massive vs. Curated Word Embeddings for Low-Resourced Languages. The Case of Yorùbá and Twi,” The International Conference on Language Resources and Evaluation (LREC 2020), Marseille, France.

11. S. Ruder, “Neural Transfer Learning for Natural Language Processing,” National University of Ireland, Galway, Ireland (2019).

12. Hal Daumé III, “Frustratingly Easy Domain Adaptation,” Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (2007), Prague, Czech Republic.

13. S. Ruder and B. Plank, “Learning to Select Data for Transfer Learning with Bayesian Optimization,” Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (2017), Copenhagen, Denmark.

14. Jing Wang, Haibo Hea Danil, and V.Prokhorov, “A Folded Neural Network Autoencoder for Dimensionality Reduction,” Procedia Computer Science 13 (2012): 120-127.

FAQ

What is the main focus of Chapter 4, “Shallow transfer learning for NLP”?Chapter 4 focuses on shallow transfer learning methods for NLP, including semisupervised learning with pretrained embeddings, multitask learning, and domain adaptation.
What does “shallow” mean in the context of transfer learning methods in this chapter?“Shallow” refers to methods that use neural networks with relatively few layers, unlike the deeper architectures discussed in later chapters.
How are pretrained word embeddings used for semisupervised learning?Pretrained word embeddings are used as feature vectors for text examples, and those vectors are then fed into traditional machine learning models such as logistic regression or random forests.
Why does the chapter revisit the IMDB sentiment classification task?The IMDB sentiment task is a standard benchmark that lets the chapter demonstrate how pretrained word embeddings and higher-level embeddings can improve classification performance.
What is the difference between word embeddings and higher-level text embeddings like sent2vec?Word embeddings represent individual words with a single vector each, while higher-level embeddings like sent2vec represent larger units such as sentences or reviews directly.
What are the main strengths of fastText mentioned in the chapter?fastText can handle out-of-vocabulary words because it uses subword character n-grams, which helps it build embeddings for unseen words.
Why does the chapter use averaging as a baseline for combining word embeddings into a review vector?Averaging is a simple and effective baseline that often works well because similar word vectors tend to stay close in embedding space, so their mean can capture overall meaning reasonably well.
What is hard-parameter sharing in multitask learning?Hard-parameter sharing means multiple tasks share some layers of the neural network, while keeping task-specific output layers separate.
What is the goal of domain adaptation in this chapter?Domain adaptation aims to adapt knowledge learned from a source domain so it can help performance in a different target domain with a different data distribution.
What is zero-shot domain adaptation, and how is it demonstrated in the chapter?Zero-shot domain adaptation means improving performance in a target domain without using labeled target-domain data for training. The chapter demonstrates this by training on IMDB reviews and then adapting to book reviews using an autoencoder.

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Transfer Learning for Natural Language Processing Transfer Learning for Natural Language Processing ebook for free
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Transfer Learning for Natural Language Processing Transfer Learning for Natural Language Processing ebook for free
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Transfer Learning for Natural Language Processing Transfer Learning for Natural Language Processing ebook for free