This chapter introduces practical text classification as an entry point into modern Natural Language Processing. It contrasts the messiness of human language with machine-designed languages, traces the field’s shift from handcrafted rules to statistical learning and deep learning, and frames classification as pattern recognition over token sequences. The narrative sets expectations: simple models can be surprisingly strong baselines, while sequence-aware neural models demand more data and compute; recent advances (culminating in Transformers) hinge on scalable architectures and massive pretraining, which the next chapters build on.
The chapter first builds a solid preprocessing pipeline: standardization (e.g., lowercasing, minimal normalization), splitting into tokens, and indexing via a learned vocabulary with special tokens like [UNK] and [PAD]. It compares character-level and word-level tokenization, then motivates subword tokenization—implemented via byte-pair encoding—to balance vocabulary size and sequence length while avoiding excessive unknowns. Throughout, it favors lightweight, framework-native tools (e.g., Keras TextVectorization) and emphasizes efficiency with dataset pipelines so tokenization on CPU doesn’t bottleneck training on accelerators. Guidance is provided on choosing tokenization based on task needs, with subword often the best default for modern models.
On IMDb sentiment analysis, the chapter walks through three modeling families. Set-based baselines discard order: a bag-of-words linear classifier with multi-hot inputs already performs strongly, and adding bigrams injects local order cues and improves accuracy further. Sequence models then encode full order: token sequences are padded and masked; an LSTM with one-hot inputs is large and slow, while switching to learned embeddings makes it far more efficient but not more accurate under limited labels. The key unlock is pretraining: a simple CBOW objective on unlabeled reviews learns a better embedding that, when transferred into the LSTM, lifts performance close to the bigram baseline. The takeaway is pragmatic: with modest resources, bag-of-words/bigram models are robust and cost-effective; with more data or stronger pretraining, sequence models gain headroom—setting the stage for Transformer-based approaches next.
The text preprocessing pipeline
Training and validation metrics for our bag of words model.
Word representations obtained from one-hot encoding or hashing are sparse, high-dimensional, and hardcoded. Word embeddings are dense, relatively low-dimensional, and learned from data.
A toy example of a word-embedding space
An Embedding layer acts as a dictionary mapping ints to floating point vectors.
The Continuous Bag of Words predicts a word based on its surrounding context with a shallow neural network.
Chapter summary
All text modeling problems involve a preprocessing step where text is broken up and transformed to integer data, called tokenization.
Tokenization can be divided into three steps. Standardization, splitting, and indexing. Standardization normalizes text, splitting breaks text up into tokens, and indexing assigns each token a unique integer ID.
There are three main types of tokenization: character, word, and subword tokenization. With an expressive enough model and enough training data, the subword tokenization is usually the most effective.
NLP models differ primarily in handling the order of input tokens.
Set models discard most order information and learn simple and fast models based solely on the presence or absence of tokens in the input. Bigram or trigram models consider the presence or absence of two or three consecutive tokens. Set models are incredibly fast to train and deploy.
Sequence models attempt to learn with the ordered sequence of tokens in the input data. Sequence models need large amounts of data to learn effectively.
An embedding is an efficient way to transform token IDs into a learned latent space. Embeddings can be trained normally with gradient descent.
Pretraining is critical for sequence models as a way to get around the data-hungry nature of these models. During pretraining an unsupervised task allows models to learn from large amounts of unlabeled text data. The learned parameters can then be transferred to a downstream task.
FAQ
What are the main steps of text preprocessing (tokenization) covered in this chapter?Tokenization typically involves three stages: (1) Standardization, where inputs are normalized (e.g., lowercasing, removing or normalizing punctuation/accents); (2) Splitting, where text is broken into tokens (characters, words, or subwords); and (3) Indexing, where tokens are mapped to integer IDs using a learned vocabulary that usually includes special tokens like [UNK] for unknowns.How do character, word, and subword tokenization compare, and when should I use each?- Character tokenization uses tiny vocabularies and never runs out of tokens, but produces long sequences.
- Word tokenization yields short sequences and is easy to interpret, but vocabularies grow very large and rare words become [UNK].
- Subword tokenization (e.g., byte-pair encoding) balances both: it keeps sequences relatively short while covering rare and misspelled words with a manageable vocabulary. Subwords are the dominant choice for modern language models.What is byte-pair encoding (BPE) and why is it useful for tokenization?BPE starts with characters and iteratively merges the most frequent adjacent symbol pairs into new symbols, building a vocabulary of subwords. It offers good compression (shorter sequences) without exploding the vocabulary size and can encode any word by backing off to smaller subwords or characters. Many modern models (including chat systems) rely on BPE-like schemes.Why are special tokens like [UNK] and [PAD] needed?[UNK] represents tokens not found in the vocabulary so any input can be indexed. [PAD] is used to pad sequences to a fixed length so batches can be processed efficiently on accelerators. Downstream sequence layers (e.g., LSTM) should ignore [PAD] positions via masking (in Keras, Embedding(mask_zero=True) handles this automatically).What is the Keras TextVectorization layer and how does it help with text classification?TextVectorization learns a vocabulary via adapt() and can:
- Output multi-hot or count vectors for set-based models (output_mode="multi_hot" or "count"),
- Produce integer token sequences for sequence models (output_mode="int"),
- Include n-grams (ngrams=2 for bigrams),
- Handle splitting and basic standardization. It integrates with tf.data for efficient on-the-fly preprocessing.How do bag-of-words and bigram models perform on IMDb sentiment classification?A simple linear classifier on multi-hot unigrams trains very fast and reaches about high-80s accuracy. Adding bigrams (ngrams=2) typically improves to around 90% with only a modest increase in parameters. Beyond trigrams, vocabulary size and sparsity explode, limiting practicality.Why did one-hot encoded LSTM underperform and run slowly compared to set-based models?One-hot encoding creates huge, sparse inputs (e.g., 600 × 30,000 per review), making training slow and memory intensive. Despite modeling order, the resulting LSTM achieved only mid–80% accuracy in this setup—worse than fast set-based models—because the representation is inefficient and the model is data-constrained.What are word embeddings and why are they better than one-hot vectors?Embeddings map tokens to dense, low-dimensional vectors learned from data. Unlike orthogonal one-hot vectors, embeddings capture semantic structure: similar words lie close together, and meaningful directions (e.g., plural, gender) emerge. They dramatically reduce input size and enable models to learn richer relationships.How can I pretrain an embedding without labels (CBOW) and why is it helpful?CBOW slides a window across text, predicting a center word from its surrounding context. Training this shallow model on unlabeled corpora (e.g., IMDb train + unsup reviews) yields a task-relevant embedding. Initializing the classifier’s Embedding layer with these weights “primes” it, improving downstream accuracy and convergence when labeled data is limited.How do I reuse a pretrained embedding in an LSTM classifier, and what gains should I expect?Create the LSTM model with an Embedding layer matching the pretrained vocabulary and dimension, assign its weights from the CBOW embedding, and train as usual (with mask_zero=True to ignore padding). In the chapter’s IMDb setup, this lifted the LSTM from mid–80% to around 89% test accuracy—competitive with the bigram model—while keeping sequence order information.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Deep Learning with Python, Third Edition ebook for free
