The Functional Neural Process

Abstract

We present a new family of exchangeable stochastic processes, the Functional Neural Processes (FNPs). FNPs model distributions over functions by learning a graph of dependencies on top of latent representations of the points in the given dataset. In doing so, they define a Bayesian model without explicitly positing a prior distribution over latent global parameters; they instead adopt priors over the relational structure of the given dataset, a task that is much simpler. We show how we can learn such models from data, demonstrate that they are scalable to large datasets through mini-batch optimization and describe how we can make predictions for new points via their posterior predictive distribution. We experimentally evaluate FNPs on the tasks of toy regression and image classification and show that, when compared to baselines that employ global latent parameters, they offer both competitive predictions as well as more robust uncertainty estimates.

arXiv commentPublished as a conference paper at NeurIPS 2019

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• Prototypical Networks for Few-shot Learning
• Categorical Reparameterization with Gumbel-Softmax
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Deep Variational Information Bottleneck
• Bayesian Compression for Deep Learning
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Functional Variational Bayesian Neural Networks
• Stochastic Variational Deep Kernel Learning
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Random Feature Expansions for Deep Gaussian Processes
• Variational Fourier features for Gaussian processes
• Recurrent Gaussian Processes
• Noisy Natural Gradient as Variational Inference
• Associative Compression Networks for Representation Learning
• Variational Implicit Processes
• Eigenvalue Corrected Noisy Natural Gradient
• Kernel Implicit Variational Inference
• Priors on exchangeable directed graphs

Cited by in corpus (15)

• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning
• Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes
• Further Analysis of Outlier Detection with Deep Generative Models
• Bootstrapping Neural Processes
• Incorporating Interpretable Output Constraints in Bayesian Neural Networks
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting
• Graph-Based Continual Learning
• ST-MAML: A Stochastic-Task based Method for Task-Heterogeneous Meta-Learning
• Energy-Based Processes for Exchangeable Data
• Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Robustifying Sequential Neural Processes
• Evidential Turing Processes