Learning Invariances in Neural Networks

Abstract

Invariances to translations have imbued convolutional neural networks with powerful generalization properties. However, we often do not know a priori what invariances are present in the data, or to what extent a model should be invariant to a given symmetry group. We show how to \emph{learn} invariances and equivariances by parameterizing a distribution over augmentations and optimizing the training loss simultaneously with respect to the network parameters and augmentation parameters. With this simple procedure we can recover the correct set and extent of invariances on image classification, regression, segmentation, and molecular property prediction from a large space of augmentations, on training data alone.

NeurIPS 2020. Code available at https://github.com/g-benton/learning-invariances

References in corpus (3)

• Improved Regularization of Convolutional Neural Networks with Cutout
• Temporal Ensembling for Semi-Supervised Learning
• Scale-Equivariant Steerable Networks

Cited by in corpus (3)

• Addressing the Topological Defects of Disentanglement via Distributed Operators
• Automated and Autonomous Experiment in Electron and Scanning Probe Microscopy
• An Empirical Framework for Domain Generalization in Clinical Settings