On the Benefits of Invariance in Neural Networks

Abstract

Many real world data analysis problems exhibit invariant structure, and models that take advantage of this structure have shown impressive empirical performance, particularly in deep learning. While the literature contains a variety of methods to incorporate invariance into models, theoretical understanding is poor and there is no way to assess when one method should be preferred over another. In this work, we analyze the benefits and limitations of two widely used approaches in deep learning in the presence of invariance: data augmentation and feature averaging. We prove that training with data augmentation leads to better estimates of risk and gradients thereof, and we provide a PAC-Bayes generalization bound for models trained with data augmentation. We also show that compared to data augmentation, feature averaging reduces generalization error when used with convex losses, and tightens PAC-Bayes bounds. We provide empirical support of these theoretical results, including a demonstration of why generalization may not improve by training with data augmentation: the `learned invariance' fails outside of the training distribution.

References in corpus (1)

• Implicit Regularization in Deep Learning

Cited by in corpus (9)

• Provably Strict Generalisation Benefit for Equivariant Models
• Causally motivated Shortcut Removal Using Auxiliary Labels
• Data augmentation and image understanding
• Understanding the Generalization Benefit of Model Invariance from a Data Perspective
• Neural Networks for Learning Counterfactual G-Invariances from Single Environments
• Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views?
• Provably Strict Generalisation Benefit for Invariance in Kernel Methods
• Implicit Bias of Linear Equivariant Networks
• A Bayesian Approach to Invariant Deep Neural Networks