12 Bayesian decision theory: Making choices under uncertainty and risks

The chapter shifts the focus of Bayesian reasoning from simply believing to deciding, showing how uncertainty and preferences must be combined to choose actions. It introduces utility as a numerical way to express how desirable an outcome is, or cost as the opposite, and uses a website launch example to illustrate how different outcomes can carry very different consequences. Rather than asking only what is most likely to happen, the framework asks which action is best on average once both probabilities and outcomes are taken into account.

The core rule is to maximize expected utility: compute the utility of each possible outcome, weight it by the probability of that outcome, and compare actions by their averages. This works for both simple discrete tables and continuous settings, where utilities are defined by functions over all possible values of an unknown quantity and expected utility is found by integration or, when that is difficult, by Monte Carlo sampling. The same principle applies across different problems, and the best choice can change when the values and costs of outcomes change.

The chapter then extends decision theory to sequential settings, where actions can be followed by new observations and further decisions. In these multi-step problems, the optimal choice depends on the full future sequence, so utilities are propagated backward through a search tree and decisions are made adaptively. This naturally reveals the tension between exploitation and exploration, and the chapter closes by describing practical approximations such as UCB, Thompson sampling, and Monte Carlo tree search, which try to balance long-term value with computational feasibility.

A tree representation of the current problem of decision making under uncertainty. Each rectangle represents a possible scenario that might take place with corresponding probabilities, while each circle represents an action we can take.

While traversing from a leaf node to an action node, we multiply the utility we obtain in the leaf node by the probability that we do obtain the utility.

After adding up the weighted utilities associated with each action, we obtain the action’s expected utility. The action with the highest expected utility is chosen by Bayesian decision theory.

Given a continuous utility function that depends on the specific value of the conversion rate 𝑟, the circle node corresponding to the decision to launch now has infinitely many branches, each yielding the utility under a particular value of 𝑟, three examples of which are shown.

The utility obtained under each action and the PDF of Beta(64, 438) as functions of the value of the conversion rate 𝑟.

The distribution of utility values of launching, induced by the samples of 𝑟. We compute the average of the distribution and compare it against the utility of not launching.

Comparison between two alternative settings of the decision problem where each conversion is valued differently. Regardless of the setup, Bayesian decision theory may be applied to determine the optimal action.

The search tree for the one-step decision-making problem, where each action’s net utility accounts for the cost of taking that action if there is one.

The search tree for a two-step decision-making problem, where after implementing a policy, we have the option to implement the remaining policy. We also have the option to terminate at any time.

A branch in the two-layer search tree, where the utility values at leaf nodes depend on which policies have been implemented, their effects, and their implementation costs.

Propagation of utility from the leaf nodes to a sub-action node, and from sub-action nodes to an intermediate state node. As sub-actions have the same expected utility, the utility of either one may be used to determine the expected utility of the previous layer’s node.

Another example of propagating utility from leaf nodes to a sub-action, and from sub-action nodes to an intermediate state node. The sub-action with the higher expected utility is chosen to have its expected utility propagated to its parent node.

Propagating the adaptive expected utilities from intermediate state nodes to the initial action node. The final number represents the expected utility of making adaptive decisions after policy A is implemented in the first stage.

Propagating utility from leaf nodes to a sub-action, and from sub-action nodes to an intermediate state node, given that policy B is implemented first, and we receive the maximum payoff. Termination is chosen as the optimal sub-action given the large payoff from the first step.

Propagating utility from leaf nodes to a sub-action, and from sub-action nodes to an intermediate state node, given that policy B is implemented first, and we receive zero payoff. Policy A should be examined next, which would help us avoid the negative terminal utility.

The upper confidence bound (UCB) heuristic with β = 2 chooses policy B as its action, agreeing with Bayesian decision theory’s decision.

• Bayesian decision theory extends Bayesian inference from what to believe to what to do.
• A utility function quantifies how desirable different outcomes are, allowing us to compare decisions under uncertainty.
• The optimal decision is the one that maximizes expected utility, which averages outcomes weighted by their probabilities.
• Discrete decision problems can be generalized to continuous ones by defining utility as a function of the unknown outcome.
• When expected utilities cannot be computed analytically, Monte Carlo sampling provides a simple and general approximation.
• In sequential settings, decisions must account for future actions and outcomes, not just immediate utility.
• The optimal strategy in a multi-step problem may differ from the optimal one-step decision due to the ability to adapt over time.
• The exploitation–exploration tradeoff arises when actions affect both immediate utility and future gains.
• Practical methods like upper confidence bound (UCB), Thompson sampling, and Monte Carlo tree search (MCTS) approximate optimal decision-making under uncertainty.

FAQ