Learning Invariances using the Marginal Likelihood

Abstract

Generalising well in supervised learning tasks relies on correctly extrapolating the training data to a large region of the input space. One way to achieve this is to constrain the predictions to be invariant to transformations on the input that are known to be irrelevant (e.g. translation). Commonly, this is done through data augmentation, where the training set is enlarged by applying hand-crafted transformations to the inputs. We argue that invariances should instead be incorporated in the model structure, and learned using the marginal likelihood, which correctly rewards the reduced complexity of invariant models. We demonstrate this for Gaussian process models, due to the ease with which their marginal likelihood can be estimated. Our main contribution is a variational inference scheme for Gaussian processes containing invariances described by a sampling procedure. We learn the sampling procedure by back-propagating through it to maximise the marginal likelihood.

Cited by in corpus (15)

• Bayesian Neural Network Priors Revisited
• Meta-Learning Symmetries by Reparameterization
• Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fix
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning
• A Bayesian Perspective on Training Speed and Model Selection
• A Parsimonious Tour of Bayesian Model Uncertainty
• Data augmentation in Bayesian neural networks and the cold posterior effect
• Bayesian Image Classification with Deep Convolutional Gaussian Processes
• Model-Based Robust Deep Learning: Generalizing to Natural, Out-of-Distribution Data
• Probabilistic Spatial Transformer Networks
• MOLPIPx: an end-to-end differentiable package for permutationally invariant polynomials in Python and Rust
• Grounding inductive biases in natural images:invariance stems from variations in data
• Learning Augmentation Distributions using Transformed Risk Minimization
• Deep Neural Networks as Point Estimates for Deep Gaussian Processes
• A Bayesian Approach to Invariant Deep Neural Networks