Learning Invariances in Neural Networks

Abstract

Invariances to translations have imbued convolutional neural networks with powerful generalization properties. However, we often do not know a priori what invariances are present in the data, or to what extent a model should be invariant to a given symmetry group. We show how to \emph{learn} invariances and equivariances by parameterizing a distribution over augmentations and optimizing the training loss simultaneously with respect to the network parameters and augmentation parameters. With this simple procedure we can recover the correct set and extent of invariances on image classification, regression, segmentation, and molecular property prediction from a large space of augmentations, on training data alone.

NeurIPS 2020. Code available at https://github.com/g-benton/learning-invariances

References in corpus (13)

• Improved Regularization of Convolutional Neural Networks with Cutout
• Fast and Accurate Modeling of Molecular Atomization Energies with Machine Learning
• Temporal Ensembling for Semi-Supervised Learning
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Rotation equivariant vector field networks
• There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
• General $E(2)$-Equivariant Steerable CNNs
• A Kernel Theory of Modern Data Augmentation
• Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data
• Scale-Equivariant Steerable Networks
• DADA: Differentiable Automatic Data Augmentation
• Oriented Response Networks
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited

Cited by in corpus (14)

• SymmetryGAN: Symmetry Discovery with Deep Learning
• Toward Decoding the Relationship between Domain Structure and Functionality in Ferroelectrics via Hidden Latent Variables
• Meta-Learning Symmetries by Reparameterization
• Automatic Symmetry Discovery with Lie Algebra Convolutional Network
• Topographic VAEs learn Equivariant Capsules
• Data augmentation in Bayesian neural networks and the cold posterior effect
• Addressing the Topological Defects of Disentanglement via Distributed Operators
• Probabilistic Spatial Transformer Networks
• Automated and Autonomous Experiment in Electron and Scanning Probe Microscopy
• Model-Based Robust Deep Learning: Generalizing to Natural, Out-of-Distribution Data
• Understanding the Generalization Benefit of Model Invariance from a Data Perspective
• Grounding inductive biases in natural images:invariance stems from variations in data
• Last Layer Marginal Likelihood for Invariance Learning
• An Empirical Framework for Domain Generalization in Clinical Settings