11 Bayesian hypothesis testing: Asking the right questions when hypothesizing
Bayesian hypothesis testing reframes the usual question from “Is the data surprising under a null hypothesis?” to “Which competing explanation best accounts for the data?” The chapter contrasts this with frequentist testing, where a p-value measures how unlikely the observed result is if the null were true, but does not directly tell us how plausible the hypothesis is or what action to take. Using a website conversion example, the text shows how classical testing can reject a baseline rate of 10% when 63 out of 500 visitors convert, yet still leave open the deeper practical question of whether the new design should actually be launched.
The Bayesian approach treats hypotheses as models and compares them directly with the Bayes factor, the ratio of their likelihoods given the observed data. In the example, a point hypothesis that the conversion rate is exactly 0.1 is compared with a broader Beta prior over possible rates, and the Bayes factor indicates how much more strongly the data support one model over the other. The chapter emphasizes that Bayesian testing requires explicitly defining the alternative model, but in return it provides a clearer measure of evidential strength, incorporates prior knowledge, and can accumulate evidence over time without the fixed-sample-size problems that can undermine p-value based testing.
The chapter also shows that Bayesian methods go beyond hypothesis comparison by supporting richer inference and decision-making. Instead of only asking whether the rate is greater than 0.1, we can ask for the posterior probability that it exceeds a business threshold, or estimate the chance of a harmful outcome. Most importantly, Bayesian reasoning connects evidence to action through expected utility: by combining posterior probabilities with gains and losses, we can decide whether launching the new design is worthwhile. This makes the framework especially valuable when the goal is not just to test a claim, but to choose the best decision under uncertainty.
The PMF of the binomial distribution representing the number of conversions out of 500 site visitors given the null hypothesis. Most probability mass centers around 50 conversions.
The right tail region of the distribution corresponds to the event that the outcome is at least as extreme as the data. The likelihood of this event gives the 𝑝-value of the hypothesis test.
Two competing hypotheses about the conversion rate 𝑟. The first hypothesis states that 𝑟 is equal to 0.1, while the second states that 𝑟 Follows a Beta(1, 1) distribution.
Diagram of Bayesian reasoning in weighing prior probabilities against evidence likelihoods to produce posteriors. The final posterior probabilities are what should inform court decisions.
Comparisons between the results from Bayesian hypothesis testing with the Bayes factor under a uniform prior (left) and a more informative prior (right). With the more informative prior, the Bayes factor offers support to both hypotheses, thus leading to less conclusive results about conversion improvement.
Left: the 𝑝-values obtained from independent binomial tests throughout six studies, only one of which shows statistical significance (in the frequentist sense). Right: the evolution of the Bayes factor comparing the constant hypothesis and a beta distribution that continually updates in light of new data. We gradually observe more evidence in support of the beta model.
Summary
- Classical hypothesis testing evaluates how surprising the data are under a null hypothesis using via a 𝑝-value.
- A 𝑝-value doesn’t tell us how likely a hypothesis is given the data, nor does it directly guide decision making.
- Bayesian hypothesis testing treats hypotheses as competing models as compares how well they explain the observed data.
- The Bayes factor measures the relative evidence for two hypotheses by comparing their marginal likelihoods.
- Unlike 𝑝-values, Bayes factors allow us to quantify evidence in favor of either hypothesis, rather than only rejecting one hypothesis.
- Bayesian methods require explicitly specifying both competing hypotheses, making assumptions more transparent.
- Incorporating prior knowledge can meaningfully affect conclusions, especially when data are limited.
- Bayesian evidence accumulates naturally over time, allowing for sequential analysis over multiple replicated experiments without starting from scratch.
- Misinterpreting conditional probabilities (such as confusing 𝑝(𝐻 | 𝐷) with 𝑝(𝐷 | 𝐻)) can lead to serious reasoning errors, as illustrated by the prosecutor’s paradox.
- Posterior distributions can provide richer information than hypothesis tests, enabling us to answer more targeted questions about outcomes.
- Decisions should be based not only on evidence, but also on consequences, which can be formalized using expected values of possible actions.
Grokking Bayes ebook for free
