Expectation Propagation for approximate Bayesian inference

Abstract

This paper presents a new deterministic approximation technique in Bayesian networks. This method, "Expectation Propagation", unifies two previous techniques: assumed-density filtering, an extension of the Kalman filter, and loopy belief propagation, an extension of belief propagation in Bayesian networks. All three algorithms try to recover an approximate distribution which is close in KL divergence to the true distribution. Loopy belief propagation, because it propagates exact belief states, is useful for a limited class of belief networks, such as those which are purely discrete. Expectation Propagation approximates the belief states by only retaining certain expectations, such as mean and variance, and iterates until these expectations are consistent throughout the network. This makes it applicable to hybrid networks with discrete and continuous nodes. Expectation Propagation also extends belief propagation in the opposite direction - it can propagate richer belief states that incorporate correlations between nodes. Experiments with Gaussian mixture models show Expectation Propagation to be convincingly better than methods with similar computational cost: Laplace's method, variational Bayes, and Monte Carlo. Expectation Propagation also provides an efficient algorithm for training Bayes point machine classifiers.

Appears in Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence (UAI2001)

References in corpus (3)

• Loopy Belief Propagation for Approximate Inference: An Empirical Study
• Tractable Inference for Complex Stochastic Processes
• A General Algorithm for Approximate Inference and its Application to Hybrid Bayes Nets

Cited by in corpus (309)

• Variational Inference: A Review for Statisticians
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• A Comparative Review of Dimension Reduction Methods in Approximate Bayesian Computation
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Optimal control as a graphical model inference problem
• Residual Belief Propagation: Informed Scheduling for Asynchronous Message Passing
• Sufficient conditions for convergence of the Sum-Product Algorithm
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Additive Gaussian Processes
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control
• Disease Progression Modeling and Prediction through Random Effect Gaussian Processes and Time Transformation
• An expert system for detecting automobile insurance fraud using social network analysis
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• EP-based Joint Active User Detection and Channel Estimation for Massive Machine-Type Communications
• Streaming Variational Bayes
• Slice sampling covariance hyperparameters of latent Gaussian models
• Variational Federated Multi-Task Learning
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• On Quadratic Penalties in Elastic Weight Consolidation
• Probabilistic Forecasting of Sensory Data with Generative Adversarial Networks - ForGAN
• On the use of marginal posteriors in marginal likelihood estimation via importance-sampling
• Bayesian Evidence and Model Selection
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Deep Probabilistic Programming
• 3D Bayesian Variational Full Waveform Inversion
• Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints
• Variational Inference using Implicit Distributions
• Skew-t Filter and Smoother with Improved Covariance Matrix Approximation
• Expectation Propagation for Continuous Time Bayesian Networks
• A Framework for Interdomain and Multioutput Gaussian Processes
• A Generalized Mean Field Algorithm for Variational Inference in Exponential Families
• Expectation propagation as a way of life: A framework for Bayesian inference on partitioned data
• Implicit Weight Uncertainty in Neural Networks
• A Theory of Solving TAP Equations for Ising Models with General Invariant Random Matrices
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• Approximate Message Passing with Unitary Transformation for Robust Bilinear Recovery
• Operator Variational Inference
• Gibbs Sampling in Factorized Continuous-Time Markov Processes
• GLASSES: Relieving The Myopia Of Bayesian Optimisation
• A Bayesian Additive Model for Understanding Public Transport Usage in Special Events
• Nonlinear Kalman Filtering with Divergence Minimization
• Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models
• Asymptotic normality and valid inference for Gaussian variational approximation
• A Factor Graph Approach to Automated Design of Bayesian Signal Processing Algorithms
• Remote Sensing Image Classification with Large Scale Gaussian Processes
• Improved Dynamic Schedules for Belief Propagation
• Parametric Return Density Estimation for Reinforcement Learning
• Neural Adaptive Sequential Monte Carlo
• The Two Kinds of Free Energy and the Bayesian Revolution
• Reasoning at the Right Time Granularity
• Gaussian Probabilities and Expectation Propagation
• Efficient Gradient-Based Inference through Transformations between Bayes Nets and Neural Nets
• Structured Region Graphs: Morphing EP into GBP
• Projection predictive model selection for Gaussian processes
• An introduction to variational inference in Geophysical inverse problems
• Measure Transformer Semantics for Bayesian Machine Learning
• Fast Gradient-Based Inference with Continuous Latent Variable Models in Auxiliary Form
• Amortized Variational Inference: A Systematic Review
• MIMO-OFDM-Based Massive Connectivity With Frequency Selectivity Compensation
• Maximum entropy and population heterogeneity in continuous cell cultures
• Practical Hilbert space approximate Bayesian Gaussian processes for probabilistic programming
• Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising
• A general computation rule for lossy summaries/messages with examples from equalization
• Advances in Variational Inference
• Approximate Inference with Amortised MCMC
• Online Bayesian Passive-Aggressive Learning
• Approximate Bayesian Computation with composite score functions
• Expectation Propagation for Nonlinear Inverse Problems -- with an Application to Electrical Impedance Tomography
• Knowledge Elicitation via Sequential Probabilistic Inference for High-Dimensional Prediction
• Fast Convergent Algorithms for Expectation Propagation Approximate Bayesian Inference
• Multi-fidelity Bayesian Optimization with Max-value Entropy Search and its parallelization
• Decision-Making with Auto-Encoding Variational Bayes
• Constructive Preference Elicitation by Setwise Max-margin Learning
• Cumulative distribution networks and the derivative-sum-product algorithm
• Join-Graph Propagation Algorithms
• Multi-Fidelity Bayesian Optimization via Deep Neural Networks
• Multi-class Gaussian Process Classification with Noisy Inputs
• Variational Gaussian Approximation for Poisson Data
• A visual introduction to Gaussian Belief Propagation
• Robust 3D reconstruction of dynamic scenes from single-photon lidar using Beta-divergences
• Approximate Bayesian Model Inversion for PDEs with Heterogeneous and State-Dependent Coefficients
• Resource-Efficient Neural Networks for Embedded Systems
• Adversarial Message Passing For Graphical Models
• Gaussian Process Vine Copulas for Multivariate Dependence
• Expectation Propagation for Neural Networks with Sparsity-promoting Priors
• A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• Sparse Linear Spectral Unmixing of Hyperspectral images using Expectation-Propagation
• Noisy-Input Entropy Search for Efficient Robust Bayesian Optimization
• Pitman-Yor Diffusion Trees
• Semi-blind Channel Estimation and Data Detection for Multi-cell Massive MIMO Systems on Time-Varying Channels
• Constrained Bayesian Optimization with Max-Value Entropy Search
• Unifying Message Passing Algorithms Under the Framework of Constrained Bethe Free Energy Minimization
• Spatio-Temporal Structured Sparse Regression with Hierarchical Gaussian Process Priors
• Bayesian computation: a perspective on the current state, and sampling backwards and forwards
• Splash: User-friendly Programming Interface for Parallelizing Stochastic Algorithms
• BayesPerf: Minimizing Performance Monitoring Errors Using Bayesian Statistics
• Solving Statistical Mechanics on Sparse Graphs with Feedback Set Variational Autoregressive Networks
• Constraining the Dynamics of Deep Probabilistic Models
• Are we Forgetting about Compositional Optimisers in Bayesian Optimisation?
• Bayesian Conditional Density Filtering
• Rigorous Dynamics of Expectation-Propagation-Based Signal Recovery from Unitarily Invariant Measurements
• A sequential Monte Carlo approach to Thompson sampling for Bayesian optimization
• PAC-Bayesian AUC classification and scoring
• Multi-objective Bayesian Optimization using Pareto-frontier Entropy
• Deep learning via message passing algorithms based on belief propagation
• Expectation Particle Belief Propagation
• A double EP-based proposal for turbo equalization
• State Space Gaussian Processes with Non-Gaussian Likelihood
• Bounding errors of Expectation-Propagation
• f-Divergence Variational Inference
• Gaussian Process Regression with a Student-t Likelihood
• On Discarding, Caching, and Recalling Samples in Active Learning
• Expectation Propagation for Poisson Data
• Virtual Vector Machine for Bayesian Online Classification
• A Generalization Bound for Online Variational Inference
• Probabilistic Sensor Fusion for Ambient Assisted Living
• Large Scale Variational Inference and Experimental Design for Sparse Generalized Linear Models
• Gaussian Process Conditional Copulas with Applications to Financial Time Series
• Relationship between fitness and heterogeneity in exponentially growing microbial populations
• Probabilistic Time of Arrival Localization
• Convergence Rates of Variational Inference in Sparse Deep Learning
• Markovian Score Climbing: Variational Inference with KL(p||q)
• On the Universality of Noiseless Linear Estimation with Respect to the Measurement Matrix
• Stochastic triangular mesh mapping: A terrain mapping technique for autonomous mobile robots
• Perturbative Corrections for Approximate Inference in Gaussian Latent Variable Models
• Infinite-dimensional gradient-based descent for alpha-divergence minimisation
• Functional Tensors for Probabilistic Programming
• Compressed sensing reconstruction using Expectation Propagation
• Posterior inference unchained with EL_2O
• Scalable Gaussian Processes on Discrete Domains
• Iterative Detection and Phase-Noise Compensation for Coded Multichannel Optical Transmission
• Belief Propagation Algorithm for Portfolio Optimization Problems
• Learning the Base Distribution in Implicit Generative Models
• Macroscopic Analysis of Vector Approximate Message Passing in a Model Mismatch Setting
• Evaluating Gaussian Process Metamodels and Sequential Designs for Noisy Level Set Estimation
• PASS-GLM: polynomial approximate sufficient statistics for scalable Bayesian GLM inference
• Adversarial $\alpha$-divergence Minimization for Bayesian Approximate Inference
• Low Complexity Receiver for Uplink SCMA System via Expectation Propagation
• Approximation by Quantization
• SAME but Different: Fast and High-Quality Gibbs Parameter Estimation
• Variational Implicit Processes
• Sparse Probit Linear Mixed Model
• On Capacity Optimality of OAMP: Beyond IID Sensing Matrices and Gaussian Signaling
• Parallel Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints
• SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models
• Multiple Kernel Learning: A Unifying Probabilistic Viewpoint
• Neural Density Estimation and Likelihood-free Inference
• Predicting Infectiousness for Proactive Contact Tracing
• Gaussian Process Quantile Regression using Expectation Propagation
• Adversarial Robustness Guarantees for Classification with Gaussian Processes
• Bayesian inference via rejection filtering
• Streaming Variational Inference for Bayesian Nonparametric Mixture Models
• Self-Averaging Expectation Propagation
• Hierarchical Gaussian Mixture Model with Objects Attached to Terminal and Non-terminal Dendrogram Nodes
• A Parsimonious Tour of Bayesian Model Uncertainty
• Discrete geometric analysis of message passing algorithm on graphs
• Scalable Multi-Class Gaussian Process Classification using Expectation Propagation
• Scalable Gaussian Process Classification via Expectation Propagation
• Chance-Constrained Active Inference
• Structured Filtering
• Neural Clustering Processes
• Effect of natural gamma background radiation on portal monitor radioisotope unmixing
• Smoothed Model Checking for Uncertain Continuous Time Markov Chains
• Conditional Variational Autoencoder for Learned Image Reconstruction
• Calibration procedures for approximate Bayesian credible sets
• Probabilistic Residual Learning for Aleatoric Uncertainty in Image Restoration
• Variational Bayesian Decision-making for Continuous Utilities
• Exploiting the Statistics of Learning and Inference
• Capacity Optimality of AMP in Coded Systems
• Fast Approximate Bayesian Computation for discretely observed Markov models using a factorised posterior distribution
• Performing Bayesian Risk Aggregation using Discrete Approximation Algorithms with Graph Factorization
• Memory Approximate Message Passing
• Bayesian Transfer Learning: An Overview of Probabilistic Graphical Models for Transfer Learning
• Nearly Consistent Finite Particle Estimates in Streaming Importance Sampling
• Adversarial Robustness Guarantees for Gaussian Processes
• Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networks
• URSABench: Comprehensive Benchmarking of Approximate Bayesian Inference Methods for Deep Neural Networks
• Mean Field Bayes Backpropagation: scalable training of multilayer neural networks with binary weights
• Spectral Method for Phase Retrieval: an Expectation Propagation Perspective
• Structured Message Passing
• On the Choice of Regions for Generalized Belief Propagation
• Inference in Multi-Layer Networks with Matrix-Valued Unknowns
• Loop corrections for approximate inference
• Tree-Structure Expectation Propagation for LDPC Decoding in Erasure Channels
• Algorithms for Joint Phase Estimation and Decoding for MIMO Systems in the Presence of Phase Noise
• What are the most important statistical ideas of the past 50 years?
• The Lazy Flipper: MAP Inference in Higher-Order Graphical Models by Depth-limited Exhaustive Search
• A Density Consistency approach to the inverse Ising problem
• A Low Complexity OFDM Receiver with Combined GAMP and MF Message Passing
• Multi-Sparse Gaussian Process: Learning based Semi-Parametric Control
• Continuously tempered Hamiltonian Monte Carlo
• A Compositional Atlas of Tractable Circuit Operations: From Simple Transformations to Complex Information-Theoretic Queries
• Variational Inference with Holder Bounds
• Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks
• How the brain might work: statistics flowing in redundant population codes
• Spatial Mapping with Gaussian Processes and Nonstationary Fourier Features
• Semi-analytic approximate stability selection for correlated data in generalized linear models
• Compressive Sensing Based Joint Activity and Data Detection for Grant-Free Massive IoT Access
• Linked Gaussian Process Emulation for Systems of Computer Models using Matérn Kernels and Adaptive Design
• Network Meta-Analysis: A Statistical Physics Perspective
• Consensus Message Passing for Layered Graphical Models
• Projecting Markov Random Field Parameters for Fast Mixing
• Bayesian Methods in Tensor Analysis
• Analysis of Random Sequential Message Passing Algorithms for Approximate Inference
• Pliable rejection sampling
• Capacity Optimal Generalized Multi-User MIMO: A Theoretical and Practical Framework
• Experimental Study on Probabilistic ToA and AoA Joint Localization in Real Indoor Environments
• Probabilistic Receiver Architecture Combining BP, MF, and EP for Multi-Signal Detection
• The Inverse G-Wishart Distribution and Variational Message Passing
• A General Framework for Constrained Bayesian Optimization using Information-based Search
• Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks
• Phase Retrieval using Expectation Consistent Signal Recovery Algorithm based on Hypernetwork
• Flexible mean field variational inference using mixtures of non-overlapping exponential families
• A Binarizing NUV Prior and its Use for M-Level Control and Digital-to-Analog Conversion
• Understanding the dynamics of message passing algorithms: a free probability heuristics
• Iterative Receivers for Downlink MIMO-SCMA: Message Passing and Distributed Cooperative Detection
• A Geometric View of Posterior Approximation
• Precision-Recall Curves Using Information Divergence Frontiers
• Variational Auto-Regressive Gaussian Processes for Continual Learning
• Modeling Social Annotation: a Bayesian Approach
• Subsampling MCMC - An introduction for the survey statistician
• Degenerate Gaussian factors for probabilistic inference
• Streaming Probabilistic Deep Tensor Factorization
• Cell-Free Massive MIMO Detection: A Distributed Expectation Propagation Approach
• A dependent partition-valued process for multitask clustering and time evolving network modelling
• Applications of Probabilistic Programming (Master's thesis, 2015)
• Energy-Based Learning for Cooperative Games, with Applications to Valuation Problems in Machine Learning
• Learning Integral Representations of Gaussian Processes
• Online Variational Filtering and Parameter Learning
• Heteroscedastic Relevance Vector Machine
• Variational Inference in Nonconjugate Models
• EigenGP: Sparse Gaussian process models with data-dependent eigenfunctions
• Sparse-posterior Gaussian Processes for general likelihoods
• Challenges and some new directions in channel coding
• Expectation Propagation performs a smoothed gradient descent
• Bayesian inference for network Poisson models
• An innovative adaptive kriging approach for efficient binary classification of mechanical problems
• Replicated Vector Approximate Message Passing For Resampling Problem
• End-to-End Probabilistic Inference for Nonstationary Audio Analysis
• A Universal Receiver for Uplink NOMA Systems
• Accelerating Psychometric Screening Tests With Bayesian Active Differential Selection
• VSEC-LDA: Boosting Topic Modeling with Embedded Vocabulary Selection
• Conditional Expectation Propagation
• Grant-Free Access via Bilinear Inference for Cell-Free MIMO with Low-Coherent Pilots
• Automating Model Comparison in Factor Graphs
• Doubly Adaptive Importance Sampling
• Inference for multiple treatment effects using confounder importance learning
• Expectation Propagation on the Maximum of Correlated Normal Variables
• Preferential Batch Bayesian Optimization
• Bayes-Optimal Convolutional AMP
• Classified Regression for Bayesian Optimization: Robot Learning with Unknown Penalties
• Efficient and Robust Machine Learning for Real-World Systems
• Equalization with Expectation Propagation at Smoothing Level
• Gaussian Process Classification with Privileged Information by Soft-to-Hard Labeling Transfer
• Approximate Marginal Posterior for Log Gaussian Cox Processes
• Principled Parallel Mean-Field Inference for Discrete Random Fields
• Time-Sensitive Bayesian Information Aggregation for Crowdsourcing Systems
• Robust EM kernel-based methods for linear system identification
• Approximate Bayesian Computation for a Class of Time Series Models
• Bayesian Inference for Gaussian Process Classifiers with Annealing and Pseudo-Marginal MCMC
• Pseudo-Marginal Bayesian Inference for Gaussian Processes
• Expectation Propagation in Gaussian Process Dynamical Systems: Extended Version
• Entropy Search for Information-Efficient Global Optimization
• Approximate inference via variational sampling
• Towards Bursting Filter Bubble via Contextual Risks and Uncertainties
• Forward Amortized Inference for Likelihood-Free Variational Marginalization
• Fast matrix computations for functional additive models
• Approximate Collapsed Gibbs Clustering with Expectation Propagation
• Probabilistic Logic Programming with Beta-Distributed Random Variables
• Mixed Variational Inference
• Variable sigma Gaussian processes: An expectation propagation perspective
• Quantile Propagation for Wasserstein-Approximate Gaussian Processes
• Bayesian Receiver Design for Grant-Free NOMA with Message Passing Based Structured Signal Estimation
• Duality between Approximate Bayesian Methods and Prior Robustness
• Combinatorial Solution of the Syndrome Decoding Problem using Copula on Grassmann graph
• Expectation propagation on the diluted Bayesian classifier
• Density Estimation via Bayesian Inference Engines
• An approach for finding fully Bayesian optimal designs using normal-based approximations to loss functions
• Sparse Algorithms for Markovian Gaussian Processes
• The Bayesian Low-Rank Determinantal Point Process Mixture Model
• Meta-Learning an Inference Algorithm for Probabilistic Programs
• Matrix completion based on Gaussian parameterized belief propagation
• Approximate Inference via Clustering
• Towards Designing Optimal Sensing Matrices for Generalized Linear Inverse Problems
• An adaptive mixture-population Monte Carlo method for likelihood-free inference
• Declarative Modeling and Bayesian Inference of Dark Matter Halos
• Nonlinear Hawkes Processes in Time-Varying System
• Probabilistic DAG Search
• Stabilization Techniques for Iterative Algorithms in Compressed Sensing
• Conditional Deep Gaussian Processes: empirical Bayes hyperdata learning
• EDML: A Method for Learning Parameters in Bayesian Networks
• A Linear Bayesian Learning Receiver Scheme for Massive MIMO Systems
• Bayesian Sampling Bias Correction: Training with the Right Loss Function
• Predictive Approaches For Gaussian Process Classifier Model Selection
• $\alpha$ Belief Propagation for Approximate Inference
• Mixed Reinforcement Learning with Additive Stochastic Uncertainty
• Bayes Networks for Supporting Query Processing Over Incomplete Autonomous Databases
• Multifidelity Bayesian Optimization for Binomial Output
• Accelerating Experimental Design by Incorporating Experimenter Hunches
• Lazy Factored Inference for Functional Probabilistic Programming
• Multimodal, high-dimensional, model-based, Bayesian inverse problems with applications in biomechanics
• An Analytically Tractable Bayesian Approximation to Optimal Point Process Filtering
• Online Low-Rank Subspace Learning from Incomplete Data: A Bayesian View
• Scalable Gaussian Processes for Supervised Hashing
• Variational Inference with Agent-Based Models
• Structured Sparse Modelling with Hierarchical GP