10 Bayesian neural networks: Bringing uncertainty to deep learning

The chapter introduces Bayesian neural networks as a way to bring uncertainty into deep learning. Standard neural networks are powerful function approximators, but they return only point predictions and give no indication of how confident they are. By placing prior distributions over weights and biases and updating them with data, a Bayesian neural network produces predictive distributions instead of single outputs, making it possible to represent both observation noise and uncertainty caused by limited data.

To build these models, the chapter frames neural network training as Bayesian inference over parameters. Because exact posterior calculation is not available for neural networks, it relies on two general-purpose approaches: MCMC, which samples from the true posterior but can be slow, and variational inference, which learns an approximate posterior more quickly by optimizing the ELBO. The chapter shows how both methods can be implemented in PyMC, and illustrates the practical tradeoff between accuracy and computational cost: MCMC tends to give higher-quality uncertainty estimates, while variational inference scales better but may produce weaker approximations if optimization is unstable or the model is too complex.

The chapter also presents dropout as a Bayesian approximation for neural networks. Originally a regularization method, dropout can be interpreted as sampling from a distribution over network architectures, and keeping dropout active at test time yields Monte Carlo samples of predictions. This MC dropout approach is lightweight and easy to integrate into existing deep learning workflows, though its uncertainty estimates are generally less accurate than those from full Bayesian inference. Overall, the chapter emphasizes that Bayesian reasoning and deep learning fit naturally together, and that the right inference method depends on the size of the problem and the level of precision required.

A sample training set where data are generated from the sine function and corrupted by noise.

The architecture of an example neural network with two hidden layers of size 6 and 4.

Predictions made by a standard neural network, which don’t include uncertainty quantification despite its importance in indicating both observation noise and data sparsity.

Parallels in non-Bayesian and Bayesian approaches to linear regression and deep learning. For each model, the non-Bayesian method finds the parameters that minimize the training error, while the Bayesian approach places a prior on each parameter and infers the posterior from data.

Predictions by a Bayesian neural network trained with MCMC. The mean line interpolates the data. The credible intervals cover most of the observed data and indicate observational noise. They also get larger in regions without observed data, indicating higher level of predictive uncertainty.

Predictions by a Bayesian neural network trained with variational inference. The predictions on the right-hand side don’t capture the downtrend of the data, illustrating the low quality of the parameters’ approximate posterior distributions.

The value of the negative ELBO as a function of the number of optimization steps. The values vary widely, indicating unstable optimization behavior.

By using more samples to approximate the ELBO’s gradient and the Adam optimizer, optimization is more stable and yields a lower loss. The trained model effectively captures the downtrend in the data.

When we don’t have access to the true observation noise level and instead model it with a half-normal distribution, the quality of the predictions by variational inference isn’t as high as that with MCMC.

Illustration of dropout. Left: a standard neural network. Right: After dropout is applied, specific nodes are randomly selected to be dropped for a given training step, producing zero output.

Each dropout step generates a new set of samples of Bernoulli variables deciding the presence of the neural network’s nodes. Each sample corresponds to a neural network drawn from a distribution of networks.

At test time, for a given input, we perform multiple forward passes with dropout enabled to obtain a distribution of predictions, which serves as our predictive distribution for that input.

Predictions by a Bayesian neural network via Monte Carlo dropout. Predictions roughly cover the observed data.

• Traditional neural networks produce point predictions by learning a single set of weights that minimize a training loss. Because their parameters are fixed after training, standard neural networks cannot quantify predictive uncertainty.
• Two forms of uncertainty arise in predictive modeling: aleatoric uncertainty, which comes from inherent noise in the data, and epistemic uncertainty, which arises from limited knowledge about the model parameters.
• A Bayesian neural network places probability distributions over the network’s weights and biases instead of learning single point estimates. Predictions by such a network are therefore probabilistic.
• Bayesian neural networks combine the expressive power of deep learning with the uncertainty quantification of Bayesian inference, enabling more reliable predictions in domains where uncertainty matters.
• Because Bayesian neural networks do not admit closed-form posteriors, inference typically relies on approximate methods.
• MCMC produces high-quality uncertainty estimates but can be computationally expensive for neural networks.
• Variational inference approximates the posterior with a parameterized distribution and learns its parameters through optimization. It scales better than MCMC but may produce lower-quality posterior approximations.
• Optimization of the variational objective (the ELBO) can be unstable in complex models like neural networks, but techniques such as additional Monte Carlo samples and adaptive optimizers like Adam can improve performance.
• Dropout, originally introduced as a regularization technique in deep learning, can be interpreted as an approximate Bayesian inference method. Specifically, dropout introduces Bernoulli random variables that randomly remove nodes during training, implicitly defining a distribution over neural network architectures.
• Keeping dropout active during prediction and performing multiple forward passes yields Monte Carlo samples from an approximate predictive distribution, a technique known as Monte Carlo dropout.
• Keeping dropout active during prediction and performing multiple forward passes yields Monte Carlo samples from an approximate predictive distribution, a technique known as Monte Carlo (MC) dropout.
• Some regularization techniques have Bayesian interpretations. For example, ridge regression corresponds to placing a Gaussian prior on model coefficients, while dropout can be interpreted as a variational approximation in a Bayesian neural network.

FAQ