This chapter introduces shallow transfer learning as a practical way to reuse pretrained knowledge in relatively simple neural and non-neural pipelines. It frames transfer learning in three main settings: cross-lingual learning, multitask learning, and domain adaptation, then focuses on the latter two while using familiar NLP classification problems as running examples. The overall goal is to show how pretrained representations and shared structure can improve performance without requiring very deep architectures.
First, the chapter explores semisupervised learning with pretrained embeddings. It revisits IMDB sentiment classification and shows how pretrained word embeddings can replace bag-of-words features, with words mapped to vectors and then aggregated into review-level features. It compares common embedding families such as word2vec, GloVe, and fastText, and then moves to higher-level representations like sent2vec and doc2vec-style ideas, where entire sentences or documents are embedded directly. Using these richer embeddings and simple downstream classifiers generally improves results over naive averaging of word vectors and can outperform a basic bag-of-words baseline.
The chapter then turns to multitask learning and domain adaptation. In multitask learning, a single shallow neural model is trained on multiple tasks at once using hard parameter sharing, so shared layers learn more general representations while task-specific heads handle each output. Although this can slightly reduce performance on individual tasks, it can yield features that transfer better to new problems. Finally, the chapter demonstrates domain adaptation by transferring an IMDB-trained sentiment model to book reviews from the Multi-Domain Sentiment Dataset. A direct zero-shot transfer already works moderately well, and an autoencoder-based projection into the source feature space yields a small but consistent improvement, illustrating how source-domain knowledge can help when target-domain labeled data is scarce or unavailable.
Visualizing the categorization of transfer learning into multitask learning, domain adaptation, and cross-lingual learning
Sequence of typical steps for semisupervised learning with word, sentence, or document embeddings
The general form of neural multitask learning we will employ—hard parameter sharing (in this case, three tasks)
The specific form of neural multitask hard parameter sharing we will employ, with two specific tasks shown—IMDB reviews and email spam classification
A visualization of the process of one-hot encoding characters into row-vector representations. The process replaces every character in the vocabulary with a sparse vector with a dimension equal to the size of the vocabulary. 1 is placed in the column corresponding to the vocabulary character index.
Summary
- Pretrained word embeddings, as well as embeddings at higher levels of text—such as sentences—are ubiquitous in NLP and can be used to convert text into numbers/vectors. This simplifies further processing to extract meaning from them.
- Such an extraction represents a form of semisupervised shallow transfer learning, which has been widely used in practice with immense success.
- Techniques like hard-parameter sharing and soft-parameter sharing allow us to create multitask learning systems, which have potential benefits, including simplified engineering design, improved generalization, and reduced overfitting.
- Sometimes it may be possible to achieve zero-shot transfer learning with no labeled data in the target domain, which is an ideal scenario because labeled data can be expensive to collect.
- It is sometimes possible to modify or adapt data in a target domain to be more similar to the data in the source domain, for instance, via projection methods such as autoencoders, and this can improve performance.
1. S. J. Pan and Q. Yang, “A Survey on Transfer Learning,” IEEE Transactions on Knowledge and Data Engineering (2009).
2. S. Ruder, “Neural Transfer Learning for Natural Language Processing,” National University of Ireland, Galway, Ireland (2019).
3. D. Wang and T. F. Zheng, “Transfer Learning for Speech and Language Processing,” Proceedings of 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA).
4. Jing Wang, Haibo Hea Danil, and V.Prokhorov, “A Folded Neural Network Autoencoder for Dimensionality Reduction,” Procedia Computer Science 13 (2012): 120-27.
5. Z. Yin and Y. Shen, “On the Dimensionality of Word Embedding,” 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montreal, Canada.
7. J. Alabi et al., “Massive vs. Curated Word Embeddings for Low-Resourced Languages. The Case of Yorùbá and Twi,” The International Conference on Language Resources and Evaluation (LREC 2020), Marseille, France.
11. S. Ruder, “Neural Transfer Learning for Natural Language Processing,” National University of Ireland, Galway, Ireland (2019).
12. Hal Daumé III, “Frustratingly Easy Domain Adaptation,” Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (2007), Prague, Czech Republic.
13. S. Ruder and B. Plank, “Learning to Select Data for Transfer Learning with Bayesian Optimization,” Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (2017), Copenhagen, Denmark.
14. Jing Wang, Haibo Hea Danil, and V.Prokhorov, “A Folded Neural Network Autoencoder for Dimensionality Reduction,” Procedia Computer Science 13 (2012): 120-127.
Transfer Learning for Natural Language Processing ebook for free