Overview
2 From effect to cause: Thomas Bayes and the first algorithm of learning
Thomas Bayes reframed probability as a way to reason backward from evidence to hidden causes and to revise beliefs as new information arrives. In place of static chance calculations, his approach created a disciplined learning process in which prior beliefs are updated into posterior probabilities through conditional probability. This shift made probability a dynamic framework for inference, and it laid the groundwork for Bayesian thinking in statistics, science, and computation.
The chapter then builds the mathematical machinery behind that framework, emphasizing that Bayes’ theorem depends on core probability rules rather than standing apart from them. The addition rule organizes mutually exclusive outcomes, the complement rule turns difficult calculations into simpler ones by considering what does not happen, and the multiplication rule connects events that occur together or in sequence. The text also highlights odds and expected value as essential tools for comparing hypotheses and making decisions under uncertainty, while warning against common mistakes such as ignoring base rates, confusing independence with exclusivity, or misreading conditional probability.
Finally, the chapter shows how repeated trials lead naturally to the binomial model and how these ideas continue to power modern machine learning and AI. Bayesian methods appear in classifiers, Bayesian networks, optimization, A/B testing, and sequential decision-making, while the underlying probability rules also shape models such as logistic regression, random forests, language models, Markov chains, and reinforcement learning. Throughout, the chapter stresses that the same timeless logic Bayes introduced centuries ago still guides learning from data today, even as the applications have become far more complex and far-reaching.
Visualizing the addition rule using a fair six-sided die. Events “rolling a 2” and “rolling a 5” are mutually exclusive, so their probabilities can be added. The shaded boxes represent the total chance of either event occurring.
Visualizing the complement rule using two six-sided dice. The shaded cells represent outcomes in which at least one die shows a 4. The unshaded cells represent the complementary event: neither die shows a 4.
Two forms of the multiplication rule. Independent events (left) combine individual probabilities because one event does not influence the next. Dependent events (right) combine the probability of the first event with the conditional probability of the second. In both cases, the probability of the complete event is obtained by multiplying probabilities along the path.
Visualizing binomial probability distributions for increasing numbers of trials (𝑛 = 10, 30, and 50), each with a success probability of 0.5. As the number of trials increases, the distributions become smoother and concentrate around the expected value, 𝑛𝑝 (dashed lines). This behavior illustrates the law of large numbers and foreshadows the close relationship between the binomial and normal distributions.
The prosecutor's fallacy arises when the probability of observing evidence if someone is innocent is confused with the probability that someone is innocent after the evidence has been observed. The outer region represents all individuals who match the evidence. The shaded region identifies those who are actually guilty, while the remaining area represents innocent individuals who also match the evidence (false positives). When the underlying event is rare, most matches may still be false positives. Bayesian inference avoids this mistake by combining the observed evidence with the prior likelihood of the event before drawing conclusions.
Summary
- Bayesian reasoning begins with the foundational rules of probability. The addition, multiplication, and complement rules provide the logical framework for quantifying uncertainty and form the basis upon which Bayes' Theorem is built. Rather than creating these principles, Bayes leveraged and combined them to develop a systematic framework for updating beliefs in light of new evidence.
- Bayes' Theorem provides a systematic method for updating beliefs in light of new evidence. By combining prior knowledge with observed data, it produces revised conclusions that remain logically consistent with the rules of probability. In doing so, it often challenges intuitive judgments by showing that compelling evidence alone may not justify a strong conclusion when the underlying event is rare or alternative explanations remain plausible.
- Thomas Bayes introduced a fundamentally new way of reasoning under uncertainty. Rather than asking how likely the observed evidence would be if a hypothesis were true, he asked how the evidence should change our confidence in the hypothesis itself. This shift from predicting observations to inferring hidden causes established a mathematical framework for learning from evidence that continues to influence statistics today.
- Bayesian inference views knowledge as something that evolves rather than remains fixed. Conclusions are continually revised as new evidence becomes available, allowing decisions to improve as understanding grows.
- Bayes' Theorem directly underlies many important methods in modern machine learning and artificial intelligence. Applications such as Naive Bayes classification, Bayesian optimization, Thompson sampling, and Bayesian A/B testing demonstrate how probabilistic learning continues to shape modern learning methods.
- The addition and multiplication rules remain essential even when Bayes' Theorem is not applied directly. These timeless principles govern probabilities across competing outcomes and related events, providing much of the probabilistic foundation for contemporary machine learning and AI.
- Bayesian reasoning reaches well beyond machine learning and artificial intelligence. From medicine and finance to engineering, public policy, and scientific research, it provides a disciplined approach for making decisions when evidence is incomplete or uncertainty cannot be eliminated.
- Bayesian inference is only as reliable as the assumptions and evidence that support it. Poorly chosen priors, biased or unrepresentative data, incorrect interpretation of probabilities, confusion between evidence and causation, and unwarranted confidence can all lead to misleading conclusions.
- More than two centuries after its publication, Bayes' work remains one of the enduring foundations of statistical reasoning. By providing a coherent framework for learning from evidence, it continues to shape both classical statistical practice and the intelligent systems that increasingly influence modern society.
Timeless Algorithms, The Foundational Ideas ebook for free
