Expectation Propagation for approximate Bayesian inference

Abstract

This paper presents a new deterministic approximation technique in Bayesian networks. This method, "Expectation Propagation", unifies two previous techniques: assumed-density filtering, an extension of the Kalman filter, and loopy belief propagation, an extension of belief propagation in Bayesian networks. All three algorithms try to recover an approximate distribution which is close in KL divergence to the true distribution. Loopy belief propagation, because it propagates exact belief states, is useful for a limited class of belief networks, such as those which are purely discrete. Expectation Propagation approximates the belief states by only retaining certain expectations, such as mean and variance, and iterates until these expectations are consistent throughout the network. This makes it applicable to hybrid networks with discrete and continuous nodes. Expectation Propagation also extends belief propagation in the opposite direction - it can propagate richer belief states that incorporate correlations between nodes. Experiments with Gaussian mixture models show Expectation Propagation to be convincingly better than methods with similar computational cost: Laplace's method, variational Bayes, and Monte Carlo. Expectation Propagation also provides an efficient algorithm for training Bayes point machine classifiers.

Appears in Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence (UAI2001)

References in corpus (3)

• Loopy Belief Propagation for Approximate Inference: An Empirical Study
• Tractable Inference for Complex Stochastic Processes
• A General Algorithm for Approximate Inference and its Application to Hybrid Bayes Nets

Cited by in corpus (13)

• An expert system for detecting automobile insurance fraud using social network analysis
• Slice sampling covariance hyperparameters of latent Gaussian models
• A Generalized Mean Field Algorithm for Variational Inference in Exponential Families
• A general computation rule for lossy summaries/messages with examples from equalization
• Fast Convergent Algorithms for Expectation Propagation Approximate Bayesian Inference
• Gaussian Process Vine Copulas for Multivariate Dependence
• Pitman-Yor Diffusion Trees
• Discrete geometric analysis of message passing algorithm on graphs
• The Lazy Flipper: MAP Inference in Higher-Order Graphical Models by Depth-limited Exhaustive Search
• Expectation Propagation on the Maximum of Correlated Normal Variables
• Heteroscedastic Relevance Vector Machine
• Variational Inference in Nonconjugate Models
• Bayes Networks for Supporting Query Processing Over Incomplete Autonomous Databases