Overview

3 Getting started with baselines: Benchmarking and optimization

This chapter builds on earlier preprocessing and baseline work by continuing two practical NLP classification examples: spam email detection and IMDB movie review sentiment analysis. Its main goal is to establish stronger benchmarks that can later be used to measure gains from transfer learning, moving beyond logistic regression and SVMs into decision-tree methods and pretrained neural language models. Along the way, it emphasizes practical comparison across models rather than relying on a single metric or a single run.

The chapter first examines decision-tree-based approaches, especially random forests and gradient-boosting machines. Random forests are presented as a fast, flexible baseline that performs well on the email task and more modestly on IMDB, while gradient boosting is shown to be more powerful but also more computationally expensive and more prone to overfitting, making cross-validation and ROC-AUC important for evaluation. It also highlights a useful advantage of tree-based methods: feature importance scores, which can help interpret what the model is learning, though these scores should be treated cautiously because they may be biased.

The chapter then introduces simple transfer learning with ELMo and BERT, where a small classifier is trained on top of each pretrained embedding model. Both require adapting the input format to the model’s expectations, and both can achieve strong results with relatively few trainable parameters, though validation curves show signs of overfitting if training continues too long. Finally, the chapter stresses that model rankings depend on hyperparameter choices, so manual and systematic tuning are essential. Increasing token limits and search ranges can improve performance across methods, and grid search over model-specific settings can yield additional gains, often without changing the overall conclusion that BERT and ELMo tend to be strongest on the harder sentiment task while simpler models remain competitive on the easier spam task.

ROC curve for the email classification example
03_01
Importance scores for the various tokens discovered by the gradient-boosting machine classifier for the IMDB classification example
03_02
Convergence of the validation and training accuracy scores for the first five epochs of training the ELMo model on the email classification example
03_03
Convergence of the validation and training accuracy scores for the first five epochs of training the ELMo model on the IMDB movie review classification example
03_04

Summary

  • It is typical to try a variety of algorithms on any given problem of interest to find the best combination of model complexity and performance for your circumstances.
  • Baselines usually start with the simplest algorithms, such as logistic regression, and become increasingly complex until the right performance/complexity trade-off is attained.
  • Important model design choices include metrics for evaluating performance, loss functions to guide the training algorithm, and best validation practices, among many others, and these can vary by model and problem type.
  • Hyperparameter tuning is an important step of the model-development pipeline, because initial hyperparameter settings may severely misrepresent the best attainable performance that can be found by tuning it.
  • Simple models tend to work best when the amount of available data isn’t very large and/or for easier problems, whereas complex neural network models tend to do better, and as such be worth the extra complexity, when more data is available.

1. P. Goyal et al., “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour,” arXhiv (2018).

FAQ

What is the main goal of the chapter on benchmarking and optimization?The chapter aims to establish solid baseline performance for two NLP tasks so later improvements from transfer learning can be measured fairly. It also introduces model comparison and hyperparameter optimization.
Which NLP problems are used as running examples in this chapter?The chapter uses two classification tasks: email spam detection and IMDB movie review sentiment classification.
Why does the chapter start with traditional machine learning baselines?Traditional methods provide a simple, fast reference point. They help reveal how difficult each problem is before moving to more advanced models like pretrained language models.
What decision-tree-based models are evaluated in this chapter?The chapter evaluates random forests and gradient-boosting machines as the main decision-tree-based baseline methods.
What is the purpose of using k-fold cross-validation with gradient-boosting machines?k-fold cross-validation gives a more reliable estimate of performance and helps detect overfitting, which is especially important because gradient-boosting machines can overfit easily.
Why is ROC AUC used in addition to accuracy for some models?ROC AUC provides a threshold-independent way to evaluate classifiers and is useful for understanding how well a model separates the classes beyond just overall accuracy.
What does feature importance tell us in decision-tree-based models?Feature importance ranks which input features contribute most to the model’s predictions. In the IMDB example, it helps identify influential words such as “worst” and “awful.”
How are ELMo and BERT used in this chapter?The chapter uses both as pretrained language models and fine-tunes only a small number of layers on the target classification tasks, demonstrating simple transfer learning for NLP.
Why are ELMo and BERT input formats different from traditional bag-of-words models?ELMo expects raw text strings, while BERT requires tokenized inputs in the form of input IDs, input masks, and segment IDs. These formats match how the pretrained models were built and trained.
What is the main takeaway from the hyperparameter tuning section?Hyperparameter tuning can improve all models by a small but meaningful amount, and it can even change which method performs best on a given task. In particular, more signal from larger token settings often improves results.

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Transfer Learning for Natural Language Processing Transfer Learning for Natural Language Processing ebook for free
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Transfer Learning for Natural Language Processing Transfer Learning for Natural Language Processing ebook for free
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Transfer Learning for Natural Language Processing Transfer Learning for Natural Language Processing ebook for free