Statistics, analytics, and machine learning are easy to misuse when people focus on getting a pleasing result instead of understanding what the data can באמת support. The chapter shows that misleading headlines, shallow interpretations, and overconfident claims often arise from ordinary human habits: we want explanations, we want success, and we are tempted to stop looking once a pattern appears. The central warning is to slow down, read past the headline, and ask whether the evidence actually matches the conclusion.
A major theme is that correlation is not causation because hidden confounding variables, biased samples, and incomplete context can create false relationships. The chapter walks through common biases such as selection bias, recall bias, anchor bias, confirmation bias, and survivorship bias, showing how each distorts what gets observed or remembered. The remedy is largely disciplined skepticism: sample more carefully, record data promptly, compare against failures as well as successes, and keep asking what else could explain the result.
The chapter also emphasizes that models and datasets operate within a limited domain, and problems arise when that domain is too broad, changes over time, or is only partially covered. In complex systems, especially AI, missing edge cases and data drift can cause performance to collapse even when a model looked strong in testing. Finally, it warns against p-hacking, the practice of hunting for a statistically significant result by trying many models, variables, or data cuts until something works. The chapter presents p-hacking as a mix of bias and incentive pressure, and argues that the best defense is honesty, preplanning, holding out fresh data, and being willing to abandon a flattering but unsupported conclusion.
Summary
Confounding variables are variables not captured in your data but that may be relevant drivers of what you observe there.
There are many types of cognitive and data biases, but in the end, bias results in information being withheld because it was never part of the selection or sampling process.
When strategizing about what data you need to sample, consider the entire operating domain of your problem, including the environment and all possible conditions your model will need to handle. This includes how conditions will change over time and cause model performance to degrade.
Be mindful of p-hacking: observing and modeling data iteratively, then cherry-picking results. Be willing and ready to find data that gives a contrary finding to the one you want, and avoid confirmation bias.
FAQ
What is the main warning of Chapter 9, “Crimes against data: Statistics done wrong”?The chapter warns that statistics, analytics, and machine learning can be misused when we try to force data to support what we want to believe. Common problems include biased sampling, confounding variables, and p-hacking, all of which can lead to misleading or false conclusions.Why should you read past a headline before trusting a statistical claim?Headlines often omit critical context or oversimplify a study’s results. A claim may sound strong in the headline, but the full article or study may reveal important caveats, such as self-reported data, observational methods, or uncontrolled variables that weaken the conclusion.What are confounding variables?Confounding variables are unmeasured factors that influence both variables you are studying, making it hard to tell what is actually causing what. They can create a false impression of correlation or hide the real cause behind an observed relationship.How can confounding variables lead to the wrong conclusion?If two variables appear correlated, you might assume one causes the other. But a third variable may be driving both, or the correlation may be due to chance. Without controlling for confounders, the conclusion may blame the wrong factor.What is data bias?Data bias happens when a sample does not adequately represent the population it is meant to describe. This can distort findings because important groups or experiences are missing, overrepresented, or underrepresented in the data.What is selection bias?Selection bias occurs when certain people or cases are more likely to end up in the sample than others. For example, polling only your own airline’s passengers about their favorite airline will likely overrepresent positive opinions about that airline.What is recall bias?Recall bias happens when people remember vivid, recent, or favorable events more easily than less memorable or unfavorable ones. This makes self-reported data unreliable, especially over long periods of time.What is anchor bias?Anchor bias is the tendency to give too much weight to the first piece of information received. In data work, this can cause early observations to influence later decisions too strongly, even when newer data suggests a different pattern.What is survivorship bias?Survivorship bias happens when you analyze only the cases that succeeded or survived and ignore the ones that failed or disappeared. This can make success look more common or easier to achieve than it really is.What is p-hacking, and why is it a problem?P-hacking is the practice of cherry-picking models, variables, data cuts, or parameters until you get a statistically significant result. It is a problem because it can create misleading findings that look real but are actually the result of repeated testing or selective reporting.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Grokking Statistics ebook for free
