4 Classification and regression
This chapter introduces practical, end-to-end neural network workflows and consolidates earlier concepts by applying them to three core problem types: binary classification, multiclass classification, and scalar regression. It walks through data preprocessing for vector inputs, simple yet effective dense architectures, activation choices, appropriate losses and metrics, optimizers, and the disciplined use of validation to detect and mitigate overfitting. A short glossary clarifies key machine-learning terms, and by the end you can build, train, evaluate, and use basic models for real-world vector data.
For classification, the chapter first tackles binary sentiment analysis on IMDb reviews using integer-encoded text vectorized into multi‑hot inputs. A small stack of dense layers with relu activations and a sigmoid output is trained with binary crossentropy and Adam; validation monitoring reveals overfitting after a few epochs, with a test accuracy around 88% and suggestions for architectural and hyperparameter experiments. It then scales to single‑label multiclass classification on Reuters newswires (46 topics), using either one‑hot or sparse labels, larger intermediate layers to avoid information bottlenecks, a softmax output with categorical crossentropy, and metrics including top‑k accuracy. Overfitting appears around nine epochs, final test accuracy is about 80%, and predictions yield full class probability distributions.
For regression, the chapter predicts California housing prices from eight numerical features. Because the dataset is small, inputs are feature‑wise normalized and targets scaled; a compact model with two relu layers and a linear output is trained with mean squared error and evaluated with mean absolute error. To obtain reliable estimates with limited data, K‑fold cross‑validation is used, yielding an average validation MAE around 0.296 (roughly $29.6k) and a final test MAE near 0.31. Core takeaways include matching output layers and loss functions to task type (sigmoid/binary crossentropy for binary, softmax/categorical crossentropy for multiclass, linear/MSE for regression), scaling inputs, monitoring overfitting via validation, and favoring smaller models and cross‑validation when data is scarce.
The three-layer model
The rectified linear unit function
The sigmoid function HTML: class=“extra-small-image”
Training and validation loss
Training and validation accuracy
Training and validation loss
Training and validation accuracy
Training and validation accuracy
3-fold cross-validation
Validation MAE by epoch
Validation MAE by epoch, excluding the first 10 data points
Chapter summary
- The three most common kinds of machine-learning tasks on vector data are binary classification, multiclass classification, and scalar regression.
- The “Wrapping up” sections earlier in the chapter summarize the important points you’ve learned regarding each task.
- Each task uses different loss functions.
-
binary_crossentropyfor binary classification. -
categorical_crossentropyfor multiclass classification. -
mean_squared_errorfor scalar regression.
-
- You’ll usually need to preprocess raw data before feeding it into a neural network.
- When your data has features with different ranges, scale each feature independently as part of preprocessing.
- As training progresses, neural networks eventually begin to overfit and obtain worse results on never-before-seen data.
- If you don’t have much training data, use a small model with only one or two intermediate layers, to avoid severe overfitting.
- If your data is divided into many categories, you may cause information bottlenecks if you make the intermediate layers too small.
- When you’re working with little data, K-fold validation can help reliably evaluate your model.
FAQ
What types of problems does Chapter 4 cover?
Binary classification (IMDb movie review sentiment), multiclass classification (Reuters newswire topics), and scalar regression (California housing prices). You’ll see complete workflows: preprocessing, model building, training, validation, and prediction.How should I preprocess text sequences for the IMDb sentiment model?
You can either pad sequences and use an Embedding layer, or multi-hot encode them into 10,000-dimensional binary vectors. The chapter uses multi-hot encoding to feed Dense layers directly.What model, activation, loss, and optimizer should I use for binary classification?
- Model: a small stack of Dense layers (e.g., 16 ReLU units ×2) and a final Dense(1, activation="sigmoid").- Loss: binary_crossentropy (best when output is a probability).
- Metric: accuracy.
- Optimizer: adam (a strong default).
Why are activation functions like ReLU necessary?
Without a nonlinearity, stacked Dense layers collapse to a single linear transformation and can’t learn complex patterns. ReLU introduces nonlinearity, enabling deep models to learn richer representations.Why use a validation set instead of the test set while tuning?
Validation results guide choices (epochs, model size, etc.). Using the test set for that would bias the final evaluation. Keep the test set untouched for the final, unbiased performance check.How do I build a multiclass classifier and encode labels?
- Architecture: end with Dense(N_classes, activation="softmax") to output a probability distribution over classes.- Label encoding and loss: use one-hot labels + categorical_crossentropy, or integer labels + sparse_categorical_crossentropy.
- Metrics: accuracy and optionally top-k accuracy (e.g., top-3).
What happens if intermediate layers are too small?
You create an information bottleneck that discards signal needed to separate many classes. In the Reuters example, reducing an intermediate layer to 4 units dropped validation accuracy by ~8%.How do I handle small datasets in regression?
- Normalize features (feature-wise mean/std from training data).- Scale targets if magnitudes are large (e.g., divide by 100,000).
- Use a small model to limit overfitting.
- Evaluate with K-fold cross-validation and track MAE/MSE.
What last-layer and loss should I use for each task type?
- Binary classification: Dense(1, sigmoid) + binary_crossentropy.- Single-label multiclass: Dense(N, softmax) + categorical_crossentropy or sparse_categorical_crossentropy.
- Scalar regression: Dense(1) with no activation (linear) + mean_squared_error (monitor MAE).
How do I interpret model.predict outputs across tasks?
- Binary classification: one probability per sample (0–1) for the positive class; choose a threshold (often 0.5).- Multiclass: a length-N vector summing to 1; argmax for the predicted class, or use top-k probabilities.
- Regression: a single continuous value (apply inverse scaling if targets were scaled).
Deep Learning with Python, Third Edition ebook for free
