7 Number go up (or down)! Correlation and linear regression

Linear regression and correlation are introduced as essential tools for understanding and predicting relationships between variables. The chapter begins by showing how a scatterplot can reveal whether two variables move together in a roughly straight-line pattern, and it emphasizes that this relationship may be positive or negative without being causal. Pearson correlation is presented as the standard numerical summary of linear association, with values ranging from -1 to 1, and as a hypothesis test that can be paired with a p-value to assess whether the observed relationship is likely to have occurred by chance.

The discussion then turns to linear regression, which fits a line to data so that one variable can be used to predict another. Using a sports drink example, the chapter explains slope, intercept, residuals, and the sum of squares as the main ingredients of fitting the best line. It shows that libraries can compute these values directly, but also walks through the underlying idea: choose the line that minimizes the squared differences between predicted and actual values. The chapter also introduces R-squared and RMSE as ways to judge how well the model captures variation in the data.

Finally, the text stresses the limits of these methods. Pearson correlation and linear regression rely on assumptions such as linearity, continuity, normality, steady variance, and the absence of extreme outliers, so plots and diagnostics are necessary before trusting the numbers. The chapter warns that correlation is not causation and that extrapolation beyond the observed data range can be misleading. It closes by broadening the view to multiple regression and a real-world bird strike example, showing that even when a pattern seems present, the relationship may be nonlinear or poorly suited to a linear model, and that good analysis requires skepticism, context, and careful validation.

A scatterplot showing 10 data points relating weather temperature to sports drink sales. The points seem to follow a “line” and have an upward trend pattern: as temperature increases, so do sales.

Fitting a line through these data points to visualize a linear relationship in the data

Different Pearson correlation values with different datasets. Note how a strong positive correlation (close to 1) or negative correlation (close to -1) has data that more closely resembles a line.

Different Pearson correlation values and p-values. Notice how the sample size and data dispersion affect the p-value.

Anscombe’s quartet, showing four different datasets that have the same correlation coefficient and linear regression, yet have very different shapes

Tyler Vigen’s database of spurious correlations includes a finding that the number of people who drowned by falling into a pool correlates with the number of films Nicolas Cage appeared in.

Using matplotlib to show a linear regression against our data

A 3D plot showing the sum of squares for different m and b values. We are trying to find the m and b values at the lowest point in this valley.

A histogram of airplane height vs. number of bird strikes

Bird strike incidents binned in 100-foot increments for HEIGHT, with a linear regression applied

Attempting a linear regression after a logarithmic transformation

• A Pearson correlation quantifies the degree to which two datasets have a linear relationship, assuming conditions such as homoscedasticity and the absence of outliers are maintained.
• A p-value can be used to quantify the statistical significance of a Pearson correlation coefficient.
• A linear regression is similar to but distinct from a Pearson correlation, as it is used to predict a dependent variable from an independent variable (like predicting sports drink sales for a given temperature).
• The least sum of squares is the most often used metric for fitting a linear regression, and understanding squared residuals is key to understanding error metrics, such as the root mean squared error (RMSE), the coefficient of determination, and the bias/variance tradeoff.
• Be mindful not to assume that a correlation means causation, and do not extrapolate trends outside the range of data you have.
• Be mindful when selecting your predictors, the variables you are hypothesizing for your model. Do not take correlation and p-value metrics at face value, and always plot your data to make sure your metrics are not misleading you.
• You can try transformations on your data to make it suitable for linear correlation or regression, but be mindful not to torture your data, and be willing to abandon your hypotheses.

FAQ