The Origins and Prevalence of Texture Bias in Convolutional Neural Networks

Abstract

Recent work has indicated that, unlike humans, ImageNet-trained CNNs tend to classify images by texture rather than by shape. How pervasive is this bias, and where does it come from? We find that, when trained on datasets of images with conflicting shape and texture, CNNs learn to classify by shape at least as easily as by texture. What factors, then, produce the texture bias in CNNs trained on ImageNet? Different unsupervised training objectives and different architectures have small but significant and largely independent effects on the level of texture bias. However, all objectives and architectures still lead to models that make texture-based classification decisions a majority of the time, even if shape information is decodable from their hidden representations. The effect of data augmentation is much larger. By taking less aggressive random crops at training time and applying simple, naturalistic augmentation (color distortion, noise, and blur), we train models that classify ambiguous images by shape a majority of the time, and outperform baselines on out-of-distribution test sets. Our results indicate that apparent differences in the way humans and ImageNet-trained CNNs process images may arise not primarily from differences in their internal workings, but from differences in the data that they see.

arXiv commentNeurIPS'2020

References in corpus (29)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Regularization of Convolutional Neural Networks with Cutout
• Shortcut Learning in Deep Neural Networks
• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition
• One weird trick for parallelizing convolutional neural networks
• Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition
• Adversarial Feature Learning
• Adversarially Learned Inference
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Certified Adversarial Robustness via Randomized Smoothing
• Generalisation in humans and deep neural networks
• Adversarial Examples Are Not Bugs, They Are Features
• Large Scale Adversarial Representation Learning
• Spatially Transformed Adversarial Examples
• Stand-Alone Self-Attention in Vision Models
• Why do deep convolutional networks generalize so poorly to small image transformations?
• Adversarial Examples that Fool both Computer Vision and Time-Limited Humans
• Adversarial Training Can Hurt Generalization
• Adversarial Examples Are a Natural Consequence of Test Error in Noise
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
• Adversarial Robustness as a Prior for Learned Representations
• Adversarial Examples Improve Image Recognition
• Are Perceptually-Aligned Gradients a General Property of Robust Classifiers?
• Online Normalization for Training Neural Networks
• A Possible Reason for why Data-Driven Beats Theory-Driven Computer Vision

Cited by in corpus (11)

• Exploiting Shape Cues for Weakly Supervised Semantic Segmentation
• Convolutional Neural Networks Rarely Learn Shape for Semantic Segmentation
• Transfer Learning Gaussian Anomaly Detection by Fine-tuning Representations
• Closer Look at the Uncertainty Estimation in Semantic Segmentation under Distributional Shift
• A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?
• Distinguishing rule- and exemplar-based generalization in learning systems
• Histogram Layers for Synthetic Aperture Sonar Imagery
• Approaching human 3D shape perception with neurally mappable models
• Clarifying Myths About the Relationship Between Shape Bias, Accuracy, and Robustness
• Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)
• Learning Less Generalizable Patterns with an Asymmetrically Trained Double Classifier for Better Test-Time Adaptation