This chapter builds on earlier preprocessing and baseline work by continuing two practical NLP classification examples: spam email detection and IMDB movie review sentiment analysis. Its main goal is to establish stronger benchmarks that can later be used to measure gains from transfer learning, moving beyond logistic regression and SVMs into decision-tree methods and pretrained neural language models. Along the way, it emphasizes practical comparison across models rather than relying on a single metric or a single run.
The chapter first examines decision-tree-based approaches, especially random forests and gradient-boosting machines. Random forests are presented as a fast, flexible baseline that performs well on the email task and more modestly on IMDB, while gradient boosting is shown to be more powerful but also more computationally expensive and more prone to overfitting, making cross-validation and ROC-AUC important for evaluation. It also highlights a useful advantage of tree-based methods: feature importance scores, which can help interpret what the model is learning, though these scores should be treated cautiously because they may be biased.
The chapter then introduces simple transfer learning with ELMo and BERT, where a small classifier is trained on top of each pretrained embedding model. Both require adapting the input format to the model’s expectations, and both can achieve strong results with relatively few trainable parameters, though validation curves show signs of overfitting if training continues too long. Finally, the chapter stresses that model rankings depend on hyperparameter choices, so manual and systematic tuning are essential. Increasing token limits and search ranges can improve performance across methods, and grid search over model-specific settings can yield additional gains, often without changing the overall conclusion that BERT and ELMo tend to be strongest on the harder sentiment task while simpler models remain competitive on the easier spam task.
ROC curve for the email classification example
Importance scores for the various tokens discovered by the gradient-boosting machine classifier for the IMDB classification example
Convergence of the validation and training accuracy scores for the first five epochs of training the ELMo model on the email classification example
Convergence of the validation and training accuracy scores for the first five epochs of training the ELMo model on the IMDB movie review classification example
Summary
- It is typical to try a variety of algorithms on any given problem of interest to find the best combination of model complexity and performance for your circumstances.
- Baselines usually start with the simplest algorithms, such as logistic regression, and become increasingly complex until the right performance/complexity trade-off is attained.
- Important model design choices include metrics for evaluating performance, loss functions to guide the training algorithm, and best validation practices, among many others, and these can vary by model and problem type.
- Hyperparameter tuning is an important step of the model-development pipeline, because initial hyperparameter settings may severely misrepresent the best attainable performance that can be found by tuning it.
- Simple models tend to work best when the amount of available data isn’t very large and/or for easier problems, whereas complex neural network models tend to do better, and as such be worth the extra complexity, when more data is available.
Transfer Learning for Natural Language Processing ebook for free