Variational Implicit Processes

Abstract

We introduce the implicit processes (IPs), a stochastic process that places implicitly defined multivariate distributions over any finite collections of random variables. IPs are therefore highly flexible implicit priors over functions, with examples including data simulators, Bayesian neural networks and non-linear transformations of stochastic processes. A novel and efficient approximate inference algorithm for IPs, namely the variational implicit processes (VIPs), is derived using generalised wake-sleep updates. This method returns simple update equations and allows scalable hyper-parameter learning with stochastic optimization. Experiments show that VIPs return better uncertainty estimates and lower errors over existing inference methods for challenging models such as Bayesian neural networks, and Gaussian processes.

arXiv commentICML 2019 camera-ready version

References in corpus (35)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Expectation Propagation for approximate Bayesian inference
• Weight Uncertainty in Neural Networks
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Deep Neural Networks as Gaussian Processes
• Structure Discovery in Nonparametric Regression through Compositional Kernel Search
• Exponential expressivity in deep neural networks through transient chaos
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Building high-level features using large scale unsupervised learning
• Amortised MAP Inference for Image Super-resolution
• Fast $\epsilon$-free Inference of Simulation Models with Bayesian Conditional Density Estimation
• Student-t Processes as Alternatives to Gaussian Processes
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks
• Functional Variational Bayesian Neural Networks
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Stochastic Variational Deep Kernel Learning
• Hierarchical Implicit Models and Likelihood-Free Variational Inference
• Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning
• Rényi Divergence Variational Inference
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• A Unifying Framework for Gaussian Process Pseudo-Point Approximations using Power Expectation Propagation
• A Spectral Approach to Gradient Estimation for Implicit Distributions
• ZhuSuan: A Library for Bayesian Deep Learning
• Improving Output Uncertainty Estimation and Generalization in Deep Learning via Neural Network Gaussian Processes
• Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs
• Approximate Inference with Amortised MCMC
• AutoGP: Exploring the Capabilities and Limitations of Gaussian Process Models
• Deep Function Machines: Generalized Neural Networks for Topological Layer Expression
• Two Methods For Wild Variational Inference
• Learning Infinite-Layer Networks: Without the Kernel Trick
• String Gaussian Process Kernels

Cited by in corpus (7)

• The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning
• Physics-informed deep generative models
• The Functional Neural Process
• Attentive Neural Processes
• Scalable Training of Inference Networks for Gaussian-Process Models
• Neural Likelihoods for Multi-Output Gaussian Processes
• Energy-Based Processes for Exchangeable Data