This chapter moves from model evaluation to model building, showing how to fit, interpret, and stress-test linear and logistic regression in R. It treats both methods as tools for prediction and for extracting actionable advice from coefficients, while emphasizing careful diagnostics, out-of-sample checks, and the dangers of extrapolation. Along the way it highlights common pitfalls such as collinearity for linear models and separation for logistic models, and motivates regularization as a principled remedy. The overall message is pragmatic: start simple, inspect assumptions and errors, validate on test data, and tune models to be both reliable and interpretable in production.
For predicting numeric outcomes, the chapter presents linear regression with lm, framing the target as a linear combination of inputs plus unsystematic noise that yields unbiased averages and good totals. Using an income example, it demonstrates training, prediction, and quality checks via true-vs-predicted plots, residual plots, R-squared, and RMSE, and it explains how to read summary(model): residual distributions, coefficient estimates and p-values, degrees of freedom, residual standard error, adjusted R-squared, and the F-test. It shows how to interpret coefficients for continuous and categorical variables (reference levels, indicator coding, intercept meaning), and warns that correlated inputs can inflate magnitudes, flip signs, and undermine coefficient-level “advice” even when predictions remain acceptable. The guidance is to trust only statistically significant coefficients, beware high standard errors and unexpected signs, and rely on test-set performance as the final safeguard.
For probabilities and binary classification, the chapter introduces logistic regression with glm (binomial logit), deriving it as a linear model on log-odds with the sigmoid mapping back to probabilities. A neonatal risk case study illustrates training, calibrated predictions (summing to observed positives), and performance analysis using score distributions, precision–recall trade-offs, enrichment, threshold selection, and confusion matrices. It covers coefficient interpretation as odds multipliers, pseudo R-squared from null and residual deviance, chi-squared tests for model significance, AIC for model choice, and Fisher scoring iterations as a convergence diagnostic; it also cautions about separation and quasi-separation. To stabilize estimates and improve generalization under collinearity or separation, the chapter introduces regularization via glmnet: ridge (L2) to “share credit,” lasso (L1) to select variables by zeroing coefficients, and elastic net to blend both, with cross-validation to choose lambda (and alpha), standardization by default, and the reminder that regularization slightly alters coefficient interpretability while often yielding safer, more robust models.
Figure 7.1. Mental model
Figure 7.2. The linear relationship between daily_cals_down and daily_exercise
Figure 7.3. pounds_lost as a linear combination of daily_exercise and daily_cals_down
Figure 7.4. Fit versus actuals for y=x^2
Figure 7.5. Building a linear model using the lm() command
Figure 7.6. Making predictions with a linear regression model
Figure 7.7. Plot of actual log income as a function of predicted log income
Figure 7.8. Plot of residual error as a function of prediction
Figure 7.9. An example of systematic errors in model predictions
Figure 7.10. The model coefficients
Figure 7.11. Model summary
Figure 7.12. Model summary coefficient columns
Figure 7.13. Mapping the odds of a flight delay to the log-odds
Figure 7.14. Mapping log-odds to the probability of a flight delay via the sigmoid function
Figure 7.15. Distribution of score broken up by positive examples (TRUE) and negative examples (FALSE)
Figure 7.16. Reproduction of the spam filter score distributions from chapter 6
Figure 7.17. Enrichment (top) and recall (bottom) plotted as a functions of threshold for the training set
Figure 7.18. Coefficients of the logistic regression model
Figure 7.19. Schematic of cv.glmnet()
Figure 7.20. Coefficients of the ridge regression model
Figure 7.21. Coefficients of the lasso regression model
Figure 7.22. Schematic of using cva.glmnet to pick alpha
Figure 7.23. Cross validation error as a function of alpha
Summary
Both linear and logistic regression assume that the outcome is a function of a linear combination of the inputs. This seems restrictive, but in practice linear and logistic regression models can perform well even when the theoretical assumptions aren’t exactly met. We’ll show how to further work around these limits in chapter 10.
Linear and logistic regression can also provide advice by quantifying the relationships between the outcomes and the model’s inputs. Since the models are expressed completely by their coefficients, they’re small, portable, and efficient—all valuable qualities when putting a model into production. If the model’s errors are uncorrelated with y, the model might be trusted to extrapolate predictions outside the training range. Extrapolation is never completely safe, but it’s sometimes necessary.
In situations where variables are correlated or the prediction problem is quasi-separable, linear methods may not perform as well. In these cases, regularization methods can produce models that are safer to apply to new data, although the coefficients of these models are not as useful for advice about the relationships between variables and the outcome.
While learning about linear models in this chapter, we have assumed that the data is well-behaved: the data had no missing values, the number of possible levels for categorical variables were low, and all possible levels were present in the training data. In real-world data, these assumptions are not always true. In the next chapter, you will learn about advanced methods to prepare ill-behaved data for modeling.
In this chapter you have learned:
- How to predict numerical quantities with linear regression models.
- How to predict probabilities or classify using logistic regression models.
- How to interpret the diagnostics from
lm()andglm()models.
- How to interpret the coefficients of linear models.
- How to diagnose when a linear model may not be "safe" or not as reliable (collinearity, quasi-separation).
- How to use
glmnetto fit regularized linear and logistic regression models.
Practical Data Science with R, Second Edition ebook for free
