Expectation Propagation for approximate Bayesian inference

Abstract

This paper presents a new deterministic approximation technique in Bayesian networks. This method, "Expectation Propagation", unifies two previous techniques: assumed-density filtering, an extension of the Kalman filter, and loopy belief propagation, an extension of belief propagation in Bayesian networks. All three algorithms try to recover an approximate distribution which is close in KL divergence to the true distribution. Loopy belief propagation, because it propagates exact belief states, is useful for a limited class of belief networks, such as those which are purely discrete. Expectation Propagation approximates the belief states by only retaining certain expectations, such as mean and variance, and iterates until these expectations are consistent throughout the network. This makes it applicable to hybrid networks with discrete and continuous nodes. Expectation Propagation also extends belief propagation in the opposite direction - it can propagate richer belief states that incorporate correlations between nodes. Experiments with Gaussian mixture models show Expectation Propagation to be convincingly better than methods with similar computational cost: Laplace's method, variational Bayes, and Monte Carlo. Expectation Propagation also provides an efficient algorithm for training Bayes point machine classifiers.

Appears in Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence (UAI2001)

References in corpus (3)

• Loopy Belief Propagation for Approximate Inference: An Empirical Study
• Tractable Inference for Complex Stochastic Processes
• A General Algorithm for Approximate Inference and its Application to Hybrid Bayes Nets

Cited by in corpus (41)

• Residual Belief Propagation: Informed Scheduling for Asynchronous Message Passing
• An expert system for detecting automobile insurance fraud using social network analysis
• Slice sampling covariance hyperparameters of latent Gaussian models
• Expectation Propagation for Continuous Time Bayesian Networks
• A Generalized Mean Field Algorithm for Variational Inference in Exponential Families
• Gibbs Sampling in Factorized Continuous-Time Markov Processes
• Asymptotic normality and valid inference for Gaussian variational approximation
• Improved Dynamic Schedules for Belief Propagation
• Parametric Return Density Estimation for Reinforcement Learning
• Reasoning at the Right Time Granularity
• Structured Region Graphs: Morphing EP into GBP
• A general computation rule for lossy summaries/messages with examples from equalization
• Fast Convergent Algorithms for Expectation Propagation Approximate Bayesian Inference
• Cumulative distribution networks and the derivative-sum-product algorithm
• Gaussian Process Vine Copulas for Multivariate Dependence
• Pitman-Yor Diffusion Trees
• Bayesian computation: a perspective on the current state, and sampling backwards and forwards
• On Discarding, Caching, and Recalling Samples in Active Learning
• PAC-Bayesian AUC classification and scoring
• Expectation Particle Belief Propagation
• Virtual Vector Machine for Bayesian Online Classification
• SAME but Different: Fast and High-Quality Gibbs Parameter Estimation
• Approximation by Quantization
• Gaussian Process Quantile Regression using Expectation Propagation
• Streaming Variational Inference for Bayesian Nonparametric Mixture Models
• Discrete geometric analysis of message passing algorithm on graphs
• Scalable Gaussian Process Classification via Expectation Propagation
• Performing Bayesian Risk Aggregation using Discrete Approximation Algorithms with Graph Factorization
• The Lazy Flipper: MAP Inference in Higher-Order Graphical Models by Depth-limited Exhaustive Search
• On the Choice of Regions for Generalized Belief Propagation
• Projecting Markov Random Field Parameters for Fast Mixing
• Consensus Message Passing for Layered Graphical Models
• Sparse-posterior Gaussian Processes for general likelihoods
• EigenGP: Sparse Gaussian process models with data-dependent eigenfunctions
• Challenges and some new directions in channel coding
• Variational Inference in Nonconjugate Models
• Heteroscedastic Relevance Vector Machine
• Expectation Propagation on the Maximum of Correlated Normal Variables
• Bayes Networks for Supporting Query Processing Over Incomplete Autonomous Databases
• EDML: A Method for Learning Parameters in Bayesian Networks
• Predictive Approaches For Gaussian Process Classifier Model Selection