Generalisation in humans and deep neural networks

Abstract

We compare the robustness of humans and current convolutional deep neural networks (DNNs) on object recognition under twelve different types of image degradations. First, using three well known DNNs (ResNet-152, VGG-19, GoogLeNet) we find the human visual system to be more robust to nearly all of the tested image manipulations, and we observe progressively diverging classification error-patterns between humans and DNNs when the signal gets weaker. Secondly, we show that DNNs trained directly on distorted images consistently surpass human performance on the exact distortion types they were trained on, yet they display extremely poor generalisation abilities when tested on other distortion types. For example, training on salt-and-pepper noise does not imply robustness on uniform white noise and vice versa. Thus, changes in the noise distribution between training and testing constitutes a crucial challenge to deep learning vision systems that can be systematically addressed in a lifelong machine learning approach. Our new dataset consisting of 83K carefully measured human psychophysical trials provide a useful reference for lifelong robustness against image degradations set by the human visual system.

Added optimal probability aggregation method to appendix

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Understanding deep learning requires rethinking generalization
• Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition
• Opening the Black Box of Deep Neural Networks via Information
• DeepGaze II: Reading fixations from deep features trained on object recognition
• Comparing deep neural networks against humans: object recognition when the signal gets weaker
• Examining the Impact of Blur on Recognition by Convolutional Networks
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
• Human perception in computer vision

Cited by in corpus (63)

• On the Opportunities and Risks of Foundation Models
• Shortcut Learning in Deep Neural Networks
• Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future
• Tent: Fully Test-time Adaptation by Entropy Minimization
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Natural Adversarial Examples
• Improving robustness against common corruptions by covariate shift adaptation
• Comparing deep neural networks against humans: object recognition when the signal gets weaker
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• The Origins and Prevalence of Texture Bias in Convolutional Neural Networks
• Controversial stimuli: pitting neural networks against each other as models of human recognition
• Benchmarking the Robustness of Semantic Segmentation Models
• Machine learning with neural networks
• The Fundamental Principles of Reproducibility
• A Fourier Perspective on Model Robustness in Computer Vision
• Test-Time Training with Self-Supervision for Generalization under Distribution Shifts
• Self-Supervised Policy Adaptation during Deployment
• An Empirical Evaluation on Robustness and Uncertainty of Regularization Methods
• Using Videos to Evaluate Image Model Robustness
• Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency
• Emergent Properties of Foveated Perceptual Systems
• The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
• On the surprising similarities between supervised and self-supervised models
• Towards Shape Biased Unsupervised Representation Learning for Domain Generalization
• On the effectiveness of adversarial training against common corruptions
• Robust Image Classification Using A Low-Pass Activation Function and DCT Augmentation
• Towards robust vision by multi-task learning on monkey visual cortex
• On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness
• Angular Visual Hardness
• Learning Online Visual Invariances for Novel Objects via Supervised and Self-Supervised Training
• Evaluating Robustness to Context-Sensitive Feature Perturbations of Different Granularities
• Overinterpretation reveals image classification model pathologies
• Guiding Visual Attention in Deep Convolutional Neural Networks Based on Human Eye Movements
• Causally motivated Shortcut Removal Using Auxiliary Labels
• Benchmarking the Robustness of Instance Segmentation Models
• The shape and simplicity biases of adversarially robust ImageNet-trained CNNs
• When Human Pose Estimation Meets Robustness: Adversarial Algorithms and Benchmarks
• Catch-Up Mix: Catch-Up Class for Struggling Filters in CNN
• MmWave Radar and Vision Fusion for Object Detection in Autonomous Driving: A Review
• Estimating the Brittleness of AI: Safety Integrity Levels and the Need for Testing Out-Of-Distribution Performance
• Rearchitecting Classification Frameworks For Increased Robustness
• A Possible Reason for why Data-Driven Beats Theory-Driven Computer Vision
• Spiking representation learning for associative memories
• HistoSPACE: Histology-Inspired Spatial Transcriptome Prediction And Characterization Engine
• Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration
• Encoding Robustness to Image Style via Adversarial Feature Perturbations
• Paradox in Deep Neural Networks: Similar yet Different while Different yet Similar
• Defending Against Image Corruptions Through Adversarial Augmentations
• Trapped in texture bias? A large scale comparison of deep instance segmentation
• Improving the robustness of ImageNet classifiers using elements of human visual cognition
• Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation
• Utilizing Network Properties to Detect Erroneous Inputs
• Estimating the Robustness of Classification Models by the Structure of the Learned Feature-Space
• Robust Training Using Natural Transformation
• Generalisation in fully-connected neural networks for time series forecasting
• Exploring Biologically Inspired Mechanisms of Adversarial Robustness
• PoisHygiene: Detecting and Mitigating Poisoning Attacks in Neural Networks
• Towards Data-Free Domain Generalization
• What Can Style Transfer and Paintings Do For Model Robustness?
• Ridge Rider: Finding Diverse Solutions by Following Eigenvectors of the Hessian
• Contextual Fusion For Adversarial Robustness
• Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs
• Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning