1 Introduction to Bayesian statistics: Representing our knowledge and uncertainty with probabilities
Bayesian statistics is introduced as a practical way to represent what we know, what we do not know, and how uncertain we are about a situation. Instead of forcing a single yes-or-no answer, it uses probability to describe beliefs about unknown quantities and to update those beliefs when new evidence appears. This makes it useful for everyday decisions like carrying an umbrella, as well as for scientific and technological problems where uncertainty is unavoidable.
The chapter builds this idea through simple weather examples that show why probabilistic thinking is more informative than deterministic predictions. A binary model can represent whether it will rain or not, while more detailed models can represent how much rain might fall. These examples lead to the idea of a random variable and a probability distribution as tools for encoding uncertainty, and they also show how expected value and cumulative probabilities summarize beliefs in a meaningful way.
The text then explains the Bayesian view more directly: probability reflects belief about an unknown quantity, a prior captures initial belief, data provide evidence, and the posterior is the updated belief after seeing that evidence. It contrasts this with frequentist probability, which treats probability as long-run frequency and focuses on the randomness of data rather than uncertainty about the unknown itself. The chapter closes by connecting Bayesian thinking to large language models and by emphasizing when Bayesian methods are especially useful, such as when prior knowledge matters, data are limited, or decisions must account for asymmetric risks.
An illustration of machine learning models without probabilistic reasoning capabilities being susceptible to noise and overconfidently making the wrong predictions.
An example categorical distribution for rainfall rate.
Summary
We need probability to model phenomena in the real world whose outcomes we haven’t observed.
With Bayesian probability, we use probability to represent our personal belief about an unknown quantity, which we model using a random variable.
From a Bayesian belief about a quantity of interest, we can compute quantities that represent our knowledge and uncertainty about that quantity of interest.
There are three main components to a Bayesian model: the prior distribution, the data, and the posterior distribution. The last component is the result of combining the first two and what we want out of a Bayesian model.
Bayesian probability is useful when we want to incorporate prior knowledge into a model, when data is limited, and for decision-making under uncertainty.
A different interpretation of probability, frequentism, views probability as the frequency of an event under infinite repeats, which limits the application of probability in various scenarios.
Large language models, which power popular chat artificial intelligence models, apply Bayesian probability to predict the next word in a sentence.
FAQ
What is Bayesian statistics in simple terms?Bayesian statistics is a way to represent what you know, what you don’t know, and how uncertain you are using probability. It helps you update your beliefs when new evidence appears.Why do we need probability to make decisions?Probability helps us reason about uncertainty instead of forcing a yes-or-no answer. This is useful when outcomes are not perfectly predictable, like deciding whether to carry an umbrella or how to respond to noisy data.How is a probabilistic model different from a non-probabilistic one?A non-probabilistic model gives a single hard answer, such as “rain” or “sunny.” A probabilistic model gives chances for different outcomes, which better reflects real-world uncertainty and supports better decisions.What is a random variable?A random variable is a variable whose value depends on the outcome of a probabilistic experiment. In this chapter, it is used to represent unknown quantities such as whether it will rain or how much it will rain.What is the difference between a binary, categorical, and continuous random variable?A binary random variable has two possible values, like 0 and 1. A categorical variable takes one value from a fixed set of categories. A continuous variable can take any value in a range, such as a rainfall rate.What is a probability distribution?A probability distribution assigns likelihoods to all possible outcomes of a random variable. In Bayesian statistics, it represents our belief about an unknown quantity and how plausible each outcome is.What is the Bernoulli distribution used for?The Bernoulli distribution models a single experiment with two outcomes, usually labeled success and failure. In the chapter, it is used to model the chance that it will rain or not rain.What are the prior, data, and posterior in a Bayesian model?The prior is your initial belief before seeing data. The data are the evidence you observe. The posterior is your updated belief after combining the prior with the data.How does Bayesian probability differ from frequentist probability?Bayesian probability treats probability as a measure of belief about an unknown quantity. Frequentist probability treats probability as the long-run frequency of outcomes under repeated trials. The two approaches answer similar questions differently.Why are large language models discussed in this chapter?LLMs are a motivating example because next-word prediction is naturally probabilistic. Bayesian ideas help explain how such models reason about multiple possible outputs, use context, and incorporate uncertainty, even though they are not fully Bayesian models.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Grokking Bayes ebook for free
