4 Optimizing the training process: Underfitting, overfitting, testing, and regularization

This chapter focuses on optimizing the training process rather than introducing a new algorithm. It explains the twin pitfalls of underfitting (models too simple to capture patterns) and overfitting (models so complex they memorize noise), using intuitive analogies and a polynomial regression example to show how the same dataset can yield each failure mode. The overarching goal is to find a model that captures the essence of the data while still generalizing well to unseen points.

To separate genuine learning from memorization, the chapter emphasizes disciplined evaluation: splitting data into training and testing sets, never touching the test set during model building (the “golden rule”), and introducing a validation set to tune hyperparameters without leakage. It presents the model complexity graph as a practical tool: as complexity increases, training error typically drops monotonically, while validation error decreases then rises, revealing the “sweet spot” that balances fit and generalization. Common split ratios (such as 80–10–10 or 60–20–20) and careful validation-driven selection help pick both model family and hyperparameters responsibly.

Regularization is introduced as a complementary strategy that bakes the simplicity preference into training by augmenting the loss with a complexity penalty. Complexity is quantified with L1 and L2 norms: adding an L1 term (lasso) encourages sparsity by driving some coefficients to exactly zero, aiding feature selection; adding an L2 term (ridge) shrinks coefficients smoothly, retaining most features but reducing variance. A tunable parameter λ governs the trade-off between accuracy and simplicity, with λ chosen via validation. Through examples (including polynomial regression), the chapter shows how regularization curbs overfitting, yields more interpretable models, and often improves performance on unseen data, along with rules of thumb for choosing between lasso and ridge in practice.

Underfitting and overfitting are two problems that can occur when training our machine learning model. Left: Underfitting happens when we oversimplify the problem at hand, and we try to solve it using a simple solution, such as trying to kill Godzilla using a fly swatter. Right: Overfitting happens when we overcomplicate the solution to a problem and try to solve it with an exceedingly complicated solution, such as trying to kill a fly using a bazooka.

In this dataset, we train some models and exhibit training problems such as underfitting and overfitting. If you were to fit a polynomial regression model to this dataset, what type of polynomial would you use: a line, a parabola, or something else?

Fitting three models to the same dataset. Model 1 is a polynomial of degree 1, which is a line. Model 2 is a polynomial of degree 2, or a quadratic. Model 3 is a polynomial of degree 10. Which one looks like the best fit?

We can use this table to decide how complex we want our model. The columns represent the three models of degree 1, 2, and 10. The rows represent the training and the testing error. The solid circles are the training set, and the white triangles are the testing set. The errors at each point can be seen as the vertical lines from the point to the curve. The error of each model is the mean absolute error given by the average of these vertical lengths. Notice that the training error goes down as the complexity of the model increases. However, the testing error goes down and then back up as the complexity increases. From this table, we conclude that out of these three models, the best one is model 2, because it gives us a low testing error.

The model complexity graph is an effective tool to help us determine the ideal complexity of a model to avoid underfitting and overfitting. In this model complexity graph, the horizontal axis represents the degree of several polynomial regression models, from 0 to 10 (i.e., the complexity of the model). The vertical axis represents the error, which in this case is given by the mean absolute error. Notice that the training error starts large and decreases as we move to the right. This is because the more complex our model is, the better it can fit the training data. The validation error, however, starts large, then decreases, and then increases again—very simple models can’t fit our data well (they underfit), whereas very complex models fit our training data but not our validation data because they overfit. A happy point in the middle is where our model neither underfits or overfits, and we can find it using the model complexity graph.

An analogy for underfitting and overfitting. Our problem consists of a broken roof. We have three roofers who can fix it. Roofer 1 comes with a bandage, roofer 2 comes with roofing shingles, and roofer 3 comes with a block of titanium. Roofer 1 oversimplified the problem, so they represent underfitting. Roofer 2 used a good solution. Roofer 3 overcomplicated the solution, so they represent overfitting.

Both L1 and L2 shrink the size of the coefficient. L1 regularization (left) does it much faster, because it subtracts a fixed amount, so it is likely to eventually become zero. L2 regularization takes much longer, because it multiplies the coefficient by a small factor, so it never reaches zero.

The dataset. Notice that its shape is a parabola that opens downward, so using linear regression won’t work well. We’ll use polynomial regression to fit this dataset, and we’ll use regularization to tune our model.

Three polynomial regression models for our dataset. The model on the left has no regularization, the model in the middle has L1 regularization with a parameter of 0.01, and the model on the right has L2 regularization with a parameter of 0.01.

FAQ