Transfer Learning for Natural Language Processing
Paul Azunre
  • MEAP began March 2020
  • Publication in Spring 2021 (estimated)
  • ISBN 9781617297267
  • 325 pages (estimated)
  • printed in black & white

A complex topic is broken down into manageable pieces, while maintaining a good pace. The text is accompanied by replicable code examples throughout.

Mathijs Affourtit
Building and training deep learning models from scratch is costly, time-consuming, and requires massive amounts of data. To address this concern, cutting-edge transfer learning techniques enable you to start with pretrained models you can tweak to meet your exact needs. In Transfer Learning for Natural Language Processing, DARPA researcher Paul Azunre takes you hands-on with customizing these open source resources for your own NLP architectures. You’ll learn how to use transfer learning to deliver state-of-the-art results even when working with limited label data, all while saving on training time and computational costs.

About the Technology

Transfer learning enables machine learning models to be initialized with existing prior knowledge. Initially pioneered in computer vision, transfer learning techniques have been revolutionising Natural Language Processing with big reductions in the training time and computation power needed for a model to start delivering results. Emerging pretrained language models such as ELMo and BERT have opened up new possibilities for NLP developers working in machine translation, semantic analysis, business analytics, and natural language generation.

About the book

Transfer Learning for Natural Language Processing is a practical primer to transfer learning techniques capable of delivering huge improvements to your NLP models. Written by DARPA researcher Paul Azunre, this practical book gets you up to speed with the relevant ML concepts before diving into the cutting-edge advances that are defining the future of NLP. You’ll learn how to adapt existing state-of-the art models into real-world applications, including building a spam email classifier, a movie review sentiment analyzer, an automated fact checker, a question-answering system and a translation system for low-resource languages.
Table of Contents detailed table of contents

Part 1: Introduction and Overview

1 What is Transfer Learning

1.1 Overview of Representative NLP Tasks

1.2 Understanding Natural Language Processing (NLP) in the Context of Artificial Intelligence (AI)

1.2.1 Artificial Intelligence (AI)

1.2.2 Machine Learning

1.2.3 Natural Language Processing (NLP)

1.3 A Brief History of NLP Advances

1.3.1 General Overview

1.3.2 Recent Transfer Learning Advances

1.4 Transfer Learning in Computer Vision

1.4.1 General Overview

1.4.2 Pre-trained ImageNet Models

1.4.3 Fine-tuning Pre-trained ImageNet Models

1.5 Why is NLP Transfer Learning an Exciting Topic to Study Now?

1.6 Summary

2 Getting Started with Baselines

2.1 Preprocessing Email Spam Classification Example Data

2.1.1 Loading and Visualizing the Enron Corpus

2.1.2 Loading and Visualizing the Fraudulent Email Corpus

2.1.3 Converting the Email Text Into Numbers

2.2 Preprocessing Movie Sentiment Classification Example Data

2.3 Generalized Linear Models

2.3.1 Logistic Regression

2.3.2 Support Vector Machines (SVMs)

2.4 Decision-Tree-Based Models

2.4.1 Random Forests (RFs)

2.4.2 Gradient Boosting Machines (GBMs)

2.5 Neural Network Models

2.5.1 Embeddings from Language Models (ELMo)

2.5.2 Bidirectional Encoder Representations from Transformers (BERT)

2.6 Optimizing Performance

2.6.1 Manual Hyperparameter Tuning

2.6.2 Systematic Hyperparameter Tuning

2.7 Summary

Part 2: Transfer Learning for NLP

3 Shallow Transfer Learning for NLP

3.1 Semi-supervised Learning with Pretrained Word Embeddings

3.2 Semi-supervised Learning with Higher-Level Representations

3.3 Multi-Task Learning

3.3.1 Problem Setup and Shallow Neural Single-Task Baseline

3.3.2 Dual Task Experiment

3.4 Domain Adaptation

3.5 Summary

4 Deep Transfer Learning for NLP with Recurrent Neural Networks

4.1 Preprocessing Tabular Column Type Classification Data

4.1.1 Obtaining and Visualizing Tabular Data

4.1.2 Preprocessing Tabular Data

4.1.3 Encoding Preprocessed Data as Numbers

4.2 Preprocessing Fact Checking Example Data

4.2.1 Special Problem Considerations

4.2.2 Loading and Visualizing Fact-Checking Data

4.3 Semantic Inference for the Modeling of Ontologies (SIMOn)

4.3.1 General Neural Architecture Overview

4.3.2 Modeling Tabular Data

4.3.3 Application of SIMOn to tabular Column Type Classification Data

4.4 Embeddings from Language Models (ELMo)

4.4.1 ELMo Bidirectional Language Modeling

4.4.2 Application to Fake News Detection

4.5 Universal Language Model Fine-Tuning (ULMFiT)

4.5.1 Target Task Language Model Fine-Tuning

4.5.2 Target Task Classifier Fine-Tuning

4.6 Summary

5 Deep Transfer Learning in NLP with Transformers

5.1 The Transformer

5.1.1 An Introduction to the Transformers Library and Attention Visualization

5.1.2 Self-Attention

5.1.3 Residual Connections, Encoder-Decoder Attention and Positional Encoding

5.1.4 Application of Pretrained Encoder-Decoder to Translation

5.2 The Generative Pretrained Transformer

5.2.1 Architecture Overview

5.2.2 Transformers Pipelines Introduction and Application to Text Generation

5.2.3 Application to Chatbots

5.3 Bidirectional Encoder Representations from Transformers (BERT)

5.3.1 Model Architecture

5.3.2 Application to Question Answering

5.3.3 Application to Fill-in-the-Blanks and Next Sentence Prediction Tasks

5.4 Cross Lingual Learning with Multilingual BERT (mBERT)

5.4.1 Brief JW300 Dataset Overview

5.4.2 Transfer mBERT to Monolingual Twi Data with Pre-trained Tokenizer

5.4.3 mBERT and Tokenizer Trained from Scratch on Monolingual Twi Data

5.5 Summary

6 Deep Transfer Learning Adaptation Strategies

7 Conclusions

Appendixes

Free GPUs in the Cloud with Kaggle Kernels and Colab Notebooks

Using Microservices to Handle Dependencies

What's inside

  • Fine tuning pretrained models with new domain data
  • Picking the right model to reduce resource usage
  • Transfer learning for neural network architectures
  • Foundations for exploring NLP academic literature

About the reader

For machine learning engineers and data scientists with some experience in NLP.

About the author

Paul Azunre holds a PhD in Computer Science from MIT and has served as a Principal Investigator on several DARPA research programs. He founded Algorine Inc., a Research Lab dedicated to advancing AI/ML and identifying scenarios where they can have a significant social impact. Paul also co-founded Ghana NLP, an open source initiative focused using NLP and Transfer Learning with Ghanaian and other low-resource languages. He frequently contributes to major peer-reviewed international research journals and serves as a program committee member at top conferences in the field.

placing your order...

Don't refresh or navigate away from the page.
Manning Early Access Program badge
Manning Early Access Program (MEAP) Read chapters as they are written, get the finished eBook as soon as it’s ready, and receive the pBook long before it's in bookstores.

5 of 7 chapters available

71% Complete
print book $37.49 $49.99 pBook + eBook + liveBook
Additional shipping charges may apply
Transfer Learning for Natural Language Processing (print book) added to cart
continue shopping
go to cart
eBook $29.99 $39.99 3 formats + liveBook
Transfer Learning for Natural Language Processing (eBook) added to cart
continue shopping
go to cart
Prices displayed in rupees will be charged in USD when you check out.
customers also bought
customers also reading
This book 1-hop 2-hops 3-hops

FREE domestic shipping on three or more pBooks

related titles