ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Abstract

Convolutional Neural Networks (CNNs) are commonly thought to recognise objects by learning increasingly complex representations of object shapes. Some recent studies suggest a more important role of image textures. We here put these conflicting hypotheses to a quantitative test by evaluating CNNs and human observers on images with a texture-shape cue conflict. We show that ImageNet-trained CNNs are strongly biased towards recognising textures rather than shapes, which is in stark contrast to human behavioural evidence and reveals fundamentally different classification strategies. We then demonstrate that the same standard architecture (ResNet-50) that learns a texture-based representation on ImageNet is able to learn a shape-based representation instead when trained on "Stylized-ImageNet", a stylized version of ImageNet. This provides a much better fit for human behavioural performance in our well-controlled psychophysical lab setting (nine experiments totalling 48,560 psychophysical trials across 97 observers) and comes with a number of unexpected emergent benefits such as improved object detection performance and previously unseen robustness towards a wide range of image distortions, highlighting advantages of a shape-based representation.

arXiv commentAccepted at ICLR 2019 (oral)

Cited by in corpus (202)

• YOLOv4: Optimal Speed and Accuracy of Object Detection
• Shortcut Learning in Deep Neural Networks
• Transfusion: Understanding Transfer Learning for Medical Imaging
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Adversarial Examples Are Not Bugs, They Are Features
• Understanding the Role of Individual Units in a Deep Neural Network
• Invariant Risk Minimization
• Tent: Fully Test-time Adaptation by Entropy Minimization
• Learning Robust Global Representations by Penalizing Local Predictive Power
• FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation
• Self-training with Noisy Student improves ImageNet classification
• What is being transferred in transfer learning?
• Natural Adversarial Examples
• Measuring Robustness to Natural Distribution Shifts in Image Classification
• Improving robustness against common corruptions by covariate shift adaptation
• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• Visual Speech Recognition for Multiple Languages in the Wild
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• The Origins and Prevalence of Texture Bias in Convolutional Neural Networks
• Controversial stimuli: pitting neural networks against each other as models of human recognition
• RobustBench: a standardized adversarial robustness benchmark
• Do Adversarially Robust ImageNet Models Transfer Better?
• A survey of synthetic data augmentation methods in computer vision
• A critical analysis of self-supervision, or what we can learn from a single image
• SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization
• AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
• Learning domain-agnostic visual representation for computational pathology using medically-irrelevant style transfer augmentation
• Adversarial Robustness as a Prior for Learned Representations
• Partial success in closing the gap between human and machine vision
• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• Domain Adaptation for Object Detection via Style Consistency
• Noise or Signal: The Role of Image Backgrounds in Object Recognition
• Reliable Fidelity and Diversity Metrics for Generative Models
• Learning With Context Feedback Loop for Robust Medical Image Segmentation
• Robust Local Features for Improving the Generalization of Adversarial Training
• Bias in Machine Learning -- What is it Good for?
• Practical Solutions for Machine Learning Safety in Autonomous Vehicles
• Adversarial Examples in Modern Machine Learning: A Review
• On the Binding Problem in Artificial Neural Networks
• Adversarial Examples Improve Image Recognition
• Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
• Learning Debiased Representation via Disentangled Feature Augmentation
• Rectifying the Shortcut Learning of Background for Few-Shot Learning
• This Looks Like That, Because ... Explaining Prototypes for Interpretable Image Recognition
• High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks
• Position, Padding and Predictions: A Deeper Look at Position Information in CNNs
• Automatic Shortcut Removal for Self-Supervised Representation Learning
• When Explanations Lie: Why Many Modified BP Attributions Fail
• SRM : A Style-based Recalibration Module for Convolutional Neural Networks
• Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformation
• Unbiased Scene Graph Generation from Biased Training
• Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data
• An Analysis of Pre-Training on Object Detection
• Zero-shot Knowledge Transfer via Adversarial Belief Matching
• Interpreting Black Box Models via Hypothesis Testing
• Adversarial score matching and improved sampling for image generation
• Counterfactual Generative Networks
• Exposing Previously Undetectable Faults in Deep Neural Networks
• RibSeg Dataset and Strong Point Cloud Baselines for Rib Segmentation from CT Scans
• Unadversarial Examples: Designing Objects for Robust Vision
• Graph Jigsaw Learning for Cartoon Face Recognition
• Towards Robust Learning-Based Pose Estimation of Noncooperative Spacecraft
• Generalizing Face Forgery Detection with High-frequency Features
• Test-time Batch Statistics Calibration for Covariate Shift
• Emergent Properties of Foveated Perceptual Systems
• Object as Hotspots: An Anchor-Free 3D Object Detection Approach via Firing of Hotspots
• Self-Supervised Variational Auto-Encoders
• Are Convolutional Neural Networks or Transformers more like human vision?
• Black-Box Ripper: Copying black-box models using generative evolutionary algorithms
• Style-transfer GANs for bridging the domain gap in synthetic pose estimator training
• BREEDS: Benchmarks for Subpopulation Shift
• On the surprising similarities between supervised and self-supervised models
• Simple black-box universal adversarial attacks on medical image classification based on deep neural networks
• Contrastive Syn-to-Real Generalization
• On the effectiveness of adversarial training against common corruptions
• Generative Adversarial Transformers
• Robust and Generalizable Visual Representation Learning via Random Convolutions
• Pruning Self-attentions into Convolutional Layers in Single Path
• Object-aware Contrastive Learning for Debiased Scene Representation
• Shape or Texture: Understanding Discriminative Features in CNNs
• 3DB: A Framework for Debugging Computer Vision Models
• Gradient Matching for Domain Generalization
• Does enhanced shape bias improve neural network robustness to common corruptions?
• Searching for Winograd-aware Quantized Networks
• WQT and DG-YOLO: towards domain generalization in underwater object detection
• Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation
• How Can CNNs Use Image Position for Segmentation?
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
• A Fourier-based Framework for Domain Generalization
• Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective
• Discrete Representations Strengthen Vision Transformer Robustness
• On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness
• What Do Deep Nets Learn? Class-wise Patterns Revealed in the Input Space
• Foreground-Aware Stylization and Consensus Pseudo-Labeling for Domain Adaptation of First-Person Hand Segmentation
• Split Batch Normalization: Improving Semi-Supervised Learning under Domain Shift
• Differentiable Drawing and Sketching
• Exploring Corruption Robustness: Inductive Biases in Vision Transformers and MLP-Mixers
• Evaluating Robustness to Context-Sensitive Feature Perturbations of Different Granularities
• Delving Deep into the Generalization of Vision Transformers under Distribution Shifts
• Overinterpretation reveals image classification model pathologies
• Do humans and Convolutional Neural Networks attend to similar areas during scene classification: Effects of task and image type
• I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively
• Image Inpainting Guided by Coherence Priors of Semantics and Textures
• Adaptive Label Smoothing
• Closer Look at the Uncertainty Estimation in Semantic Segmentation under Distributional Shift
• Random Noise Defense Against Query-Based Black-Box Attacks
• When Human Pose Estimation Meets Robustness: Adversarial Algorithms and Benchmarks
• Assessing Robustness of Deep learning Methods in Dermatological Workflow
• The shape and simplicity biases of adversarially robust ImageNet-trained CNNs
• Verification of Size Invariance in DNN Activations using Concept Embeddings
• Transforming Neural Network Visual Representations to Predict Human Judgments of Similarity
• Increasing Shape Bias in ImageNet-Trained Networks Using Transfer Learning and Domain-Adversarial Methods
• Temporal Shuffling for Defending Deep Action Recognition Models against Adversarial Attacks
• Texture Bias Of CNNs Limits Few-Shot Classification Performance
• Robust Semantic Segmentation with Superpixel-Mix
• Toward Learning Human-aligned Cross-domain Robust Models by Countering Misaligned Features
• Robust and Accurate Object Detection via Adversarial Learning
• Finding Generalizable Evidence by Learning to Convince Q&A Models
• TRIER: Template-Guided Neural Networks for Robust and Interpretable Sleep Stage Identification from EEG Recordings
• Robustness in Compressed Neural Networks for Object Detection
• CLCC: Contrastive Learning for Color Constancy
• Shape-Texture Debiased Neural Network Training
• Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples
• Recent Advances in Understanding Adversarial Robustness of Deep Neural Networks
• A Self-Supervised Feature Map Augmentation (FMA) Loss and Combined Augmentations Finetuning to Efficiently Improve the Robustness of CNNs
• Hierarchically Compositional Tasks and Deep Convolutional Networks
• Learning to Predict Trustworthiness with Steep Slope Loss
• SWAD: Domain Generalization by Seeking Flat Minima
• Pyramid Adversarial Training Improves ViT Performance
• Generative Interventions for Causal Learning
• A Fine-Grained Analysis on Distribution Shift
• BiaSwap: Removing dataset bias with bias-tailored swapping augmentation
• Universal Adversarial Perturbations Through the Lens of Deep Steganography: Towards A Fourier Perspective
• SketchEmbedNet: Learning Novel Concepts by Imitating Drawings
• Large-Scale Historical Watermark Recognition: dataset and a new consistency-based approach
• Repaint: Improving the Generalization of Down-Stream Visual Tasks by Generating Multiple Instances of Training Examples
• Encoding Robustness to Image Style via Adversarial Feature Perturbations
• Synthesis in Style: Semantic Segmentation of Historical Documents using Synthetic Data
• Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration
• ConvNets for Counting: Object Detection of Transient Phenomena in Steelpan Drums
• StyleAugment: Learning Texture De-biased Representations by Style Augmentation without Pre-defined Textures
• A neural anisotropic view of underspecification in deep learning
• Exploring and Distilling Cross-Modal Information for Image Captioning
• Learning compact generalizable neural representations supporting perceptual grouping
• Defending Against Image Corruptions Through Adversarial Augmentations
• SPEED+: Next-Generation Dataset for Spacecraft Pose Estimation across Domain Gap
• Overcoming Statistical Shortcuts for Open-ended Visual Counting
• Bridging In- and Out-of-distribution Samples for Their Better Discriminability
• Deep Learning of Unified Region, Edge, and Contour Models for Automated Image Segmentation
• Artistic Domain Generalisation Methods are Limited by their Deep Representations
• Visual Privacy Protection via Mapping Distortion
• Towards Human-Understandable Visual Explanations:Imperceptible High-frequency Cues Can Better Be Removed
• Switched linear projections for neural network interpretability
• SCNet: A Generalized Attention-based Model for Crack Fault Segmentation
• Revisiting Edge Detection in Convolutional Neural Networks
• What Do Adversarially Robust Models Look At?
• Robust Classification using Robust Feature Augmentation
• Histogram Matching Augmentation for Domain Adaptation with Application to Multi-Centre, Multi-Vendor and Multi-Disease Cardiac Image Segmentation
• Natural Adversarial Objects
• Untapped Potential of Data Augmentation: A Domain Generalization Viewpoint
• Encouraging Intra-Class Diversity Through a Reverse Contrastive Loss for Better Single-Source Domain Generalization
• Towards Robust Classification Model by Counterfactual and Invariant Data Generation
• A Procedural World Generation Framework for Systematic Evaluation of Continual Learning
• A Study for Universal Adversarial Attacks on Texture Recognition
• Real-time Detection of Practical Universal Adversarial Perturbations
• A Novel Measure to Evaluate Generative Adversarial Networks Based on Direct Analysis of Generated Images
• Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)
• Emotion Transfer Using Vector-Valued Infinite Task Learning
• Mind the Gap: On Bridging the Semantic Gap between Machine Learning and Information Security
• On Exploring and Improving Robustness of Scene Text Detection Models
• Meta-learning Transferable Representations with a Single Target Domain
• Unsupervised Learning of Debiased Representations with Pseudo-Attributes
• Structure-Preserving Progressive Low-rank Image Completion for Defending Adversarial Attacks
• Synthesizing human-like sketches from natural images using a conditional convolutional decoder
• Analyzing the Dependency of ConvNets on Spatial Information
• ProtoShotXAI: Using Prototypical Few-Shot Architecture for Explainable AI
• Combining Diverse Feature Priors
• Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs
• IFBiD: Inference-Free Bias Detection
• Fighting Fire with Fire: Contrastive Debiasing without Bias-free Data via Generative Bias-transformation
• Counterfactual Supervision-based Information Bottleneck for Out-of-Distribution Generalization
• White Paper Assistance: A Step Forward Beyond the Shortcut Learning
• RC2020 Report: Learning De-biased Representations with Biased Representations
• Mixing-AdaSIN: Constructing a De-biased Dataset using Adaptive Structural Instance Normalization and Texture Mixing
• Understanding top-down attention using task-oriented ablation design
• Identifying Layers Susceptible to Adversarial Attacks
• Comparing the Machine Readability of Traffic Sign Pictograms in Austria and Germany
• DessiLBI: Exploring Structural Sparsity of Deep Networks via Differential Inclusion Paths
• Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Time
• Extreme Value Preserving Networks
• Assessing The Importance Of Colours For CNNs In Object Recognition
• Understanding the robustness of deep neural network classifiers for breast cancer screening
• Combating Unknown Bias with Effective Bias-Conflicting Scoring and Gradient Alignment
• CleftNet: Augmented Deep Learning for Synaptic Cleft Detection from Brain Electron Microscopy
• Mining Domain Knowledge: Improved Framework towards Automatically Standardizing Anatomical Structure Nomenclature in Radiotherapy
• Progressive Compressed Records: Taking a Byte out of Deep Learning Data
• Can Targeted Adversarial Examples Transfer When the Source and Target Models Have No Label Space Overlap?
• Describing Textures using Natural Language
• Deep Flow Collaborative Network for Online Visual Tracking
• RGB-D Individual Segmentation
• Generating Object Stamps
• Multi-Glimpse Network: A Robust and Efficient Classification Architecture based on Recurrent Downsampled Attention