The Pitfalls of Simplicity Bias in Neural Networks

Abstract

Several works have proposed Simplicity Bias (SB)---the tendency of standard training procedures such as Stochastic Gradient Descent (SGD) to find simple models---to justify why neural networks generalize well [Arpit et al. 2017, Nakkiran et al. 2019, Soudry et al. 2018]. However, the precise notion of simplicity remains vague. Furthermore, previous settings that use SB to theoretically justify why neural networks generalize well do not simultaneously capture the non-robustness of neural networks---a widely observed phenomenon in practice [Goodfellow et al. 2014, Jo and Bengio 2017]. We attempt to reconcile SB and the superior standard generalization of neural networks with the non-robustness observed in practice by designing datasets that (a) incorporate a precise notion of simplicity, (b) comprise multiple predictive features with varying levels of simplicity, and (c) capture the non-robustness of neural networks trained on real data. Through theory and empirics on these datasets, we make four observations: (i) SB of SGD and variants can be extreme: neural networks can exclusively rely on the simplest feature and remain invariant to all predictive complex features. (ii) The extreme aspect of SB could explain why seemingly benign distribution shifts and small adversarial perturbations significantly degrade model performance. (iii) Contrary to conventional wisdom, SB can also hurt generalization on the same data distribution, as SB persists even when the simplest feature has less predictive power than the more complex features. (iv) Common approaches to improve generalization and robustness---ensembles and adversarial training---can fail in mitigating SB and its pitfalls. Given the role of SB in training neural networks, we hope that the proposed datasets and methods serve as an effective testbed to evaluate novel algorithmic approaches aimed at avoiding the pitfalls of SB.

arXiv commentNeurIPS 2020

References in corpus (38)

• Adam: A Method for Stochastic Optimization
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Densely Connected Convolutional Networks
• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• On Calibration of Modern Neural Networks
• Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
• Theoretically Principled Trade-off between Robustness and Accuracy
• Poisoning Attacks against Support Vector Machines
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Adversarially Robust Generalization Requires More Data
• Fast is better than free: Revisiting adversarial training
• The Space of Transferable Adversarial Examples
• A Closer Look at Memorization in Deep Networks
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Learning Robust Global Representations by Penalizing Local Predictive Power
• Exploring the Landscape of Spatial Robustness
• On Lazy Training in Differentiable Programming
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Improving Adversarial Robustness via Promoting Ensemble Diversity
• Natural Adversarial Examples
• Adversarial examples from computational constraints
• Uniform convergence may be unable to explain generalization in deep learning
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Improving Adversarial Robustness of Ensembles with Diversity Training
• Learning Robust Representations by Projecting Superficial Statistics Out
• Ensemble Methods as a Defense to Adversarial Perturbations Against Deep Neural Networks
• Unrestricted Adversarial Examples via Semantic Manipulation
• Failures of Gradient-Based Deep Learning
• Kernel and Rich Regimes in Overparametrized Models
• SGD on Neural Networks Learns Functions of Increasing Complexity
• SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
• Intriguing properties of adversarial training at scale
• Computational Limitations in Robust Classification and Win-Win Results

Cited by in corpus (16)

• Shortcut Learning in Deep Neural Networks
• Domain Generalization using Causal Matching
• Intriguing Properties of Contrastive Losses
• What shapes feature representations? Exploring datasets, architectures, and training
• Gradient Starvation: A Learning Proclivity in Neural Networks
• The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning
• Understanding Deep Learning via Decision Boundary
• Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
• Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Models
• The Low-Rank Simplicity Bias in Deep Networks
• Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition
• Simple data balancing achieves competitive worst-group-accuracy
• Simplicity Bias Leads to Amplified Performance Disparities
• The Connection between Out-of-Distribution Generalization and Privacy of ML Models
• Echoes: Unsupervised Debiasing via Pseudo-bias Labeling in an Echo Chamber
• Learning Less Generalizable Patterns with an Asymmetrically Trained Double Classifier for Better Test-Time Adaptation