1 Introduction to Bayesian statistics: Representing our knowledge and uncertainty with probabilities
Bayesian statistics is presented as a practical way to reason about uncertainty, make predictions, and update beliefs when new evidence appears. Rather than forcing a single answer, it represents unknown quantities with probability distributions, letting us capture both what we think is most likely and how confident we are. This makes it especially useful in situations where decisions must be made under risk, with noisy data, or with limited information.
The chapter explains why probability matters by using weather forecasting as a familiar example. A simple yes-or-no prediction ignores uncertainty, while a probabilistic forecast can express different possible outcomes and support better decisions, such as whether to bring an umbrella. It then introduces random variables, Bernoulli and categorical distributions, expected value, and the idea of granularity, showing how Bayesian reasoning can describe not just whether something happens, but how likely different outcomes are.
It also contrasts Bayesian and frequentist interpretations of probability: Bayesian methods treat probability as belief about an unknown quantity, while frequentist methods view it as long-run frequency under repeated trials. Bayesian models combine a prior belief, observed data, and a posterior updated belief, which can be especially valuable when data are scarce or prior knowledge matters. The chapter closes by noting that these ideas are relevant in modern AI, including large language models, and by previewing how the book will develop from intuitive foundations to more advanced Bayesian methods and decision-making tools.
An illustration of machine learning models without probabilistic reasoning capabilities being susceptible to noise and overconfidently making the wrong predictions.
An example categorical distribution for rainfall rate.
Summary
We need probability to model phenomena in the real world whose outcomes we haven’t observed.
With Bayesian probability, we use probability to represent our personal belief about an unknown quantity, which we model using a random variable.
From a Bayesian belief about a quantity of interest, we can compute quantities that represent our knowledge and uncertainty about that quantity of interest.
There are three main components to a Bayesian model: the prior distribution, the data, and the posterior distribution. The last component is the result of combining the first two and what we want out of a Bayesian model.
Bayesian probability is useful when we want to incorporate prior knowledge into a model, when data is limited, and for decision-making under uncertainty.
A different interpretation of probability, frequentism, views probability as the frequency of an event under infinite repeats, which limits the application of probability in various scenarios.
Large language models, which power popular chat artificial intelligence models, apply Bayesian probability to predict the next word in a sentence.
FAQ
What is Bayesian statistics in simple terms?Bayesian statistics is a way to make predictions and decisions under uncertainty. It lets you represent what you currently believe about an unknown quantity, then update that belief when new data or evidence arrives.Why do we need probability instead of only yes-or-no predictions?Real-world predictions are rarely perfectly accurate. Probability lets us express uncertainty, measure how likely different outcomes are, and make better decisions than a simple “yes” or “no” answer.What is a random variable?A random variable is a variable whose value depends on the outcome of a probabilistic experiment. In the chapter’s weather example, it can represent whether it rains today or how much rain falls.What is the difference between binary, categorical, and continuous random variables?A binary random variable has two possible values, such as rain or no rain. A categorical variable can take one of a fixed set of categories. A continuous variable can take any value in a range, such as a rainfall amount measured on a scale.What is a probability distribution?A probability distribution describes the likelihood of possible outcomes for a random variable. It tells you how probable each value or range of values is.What are the main parts of a Bayesian model?The three main parts are the prior distribution, which represents your initial belief; the data, which is the evidence you observe; and the posterior distribution, which is your updated belief after seeing the data.What is the difference between the prior and the posterior?The prior is your belief before seeing any data. The posterior is the updated belief after incorporating the data. In Bayesian thinking, learning is the process of moving from prior to posterior.How is Bayesian probability different from frequentist probability?Bayesian probability treats probability as a degree of belief about an unknown quantity. Frequentist probability treats probability as the long-run frequency of an event under repeated trials. Bayesian methods use prior knowledge; frequentist methods focus more on data alone.Why are Bayesian methods useful when data is limited or noisy?Bayesian methods can combine prior knowledge with limited data, which helps when observations are sparse or noisy. They also produce richer outputs, like full probability distributions, rather than just a single estimate.How are large language models related to Bayesian probability?LLMs can be viewed as probabilistic next-word predictors. Bayesian thinking helps describe how they use context and data to assign likelihoods to possible next words, and it also relates to how they can be refined with feedback.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Grokking Bayes ebook for free
