Partial success in closing the gap between human and machine vision

Abstract

A few years ago, the first CNN surpassed human performance on ImageNet. However, it soon became clear that machines lack robustness on more challenging test cases, a major obstacle towards deploying machines "in the wild" and towards obtaining better computational models of human visual perception. Here we ask: Are we making progress in closing the gap between human and machine vision? To answer this question, we tested human observers on a broad range of out-of-distribution (OOD) datasets, recording 85,120 psychophysical trials across 90 participants. We then investigated a range of promising machine learning developments that crucially deviate from standard supervised CNNs along three axes: objective function (self-supervised, adversarially trained, CLIP language-image training), architecture (e.g. vision transformers), and dataset size (ranging from 1M to 1B). Our findings are threefold. (1.) The longstanding distortion robustness gap between humans and CNNs is closing, with the best models now exceeding human feedforward performance on most of the investigated OOD datasets. (2.) There is still a substantial image-level consistency gap, meaning that humans make different errors than models. In contrast, most models systematically agree in their categorisation errors, even substantially different ones like contrastive self-supervised vs. standard supervised models. (3.) In many cases, human-to-model consistency improves when training dataset size is increased by one to three orders of magnitude. Our results give reason for cautious optimism: While there is still much room for improvement, the behavioural difference between human and machine vision is narrowing. In order to measure future progress, 17 OOD datasets with image-level human behavioural data and evaluation code are provided as a toolbox and benchmark at: https://github.com/bethgelab/model-vs-human/

NeurIPS 2021 Oral, camera ready version. A preliminary version of this work was presented as Oral at the 2020 NeurIPS workshop on "Shared Visual Representations in Human & Machine Intelligence" (arXiv:2010.08377)

References in corpus (35)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Explaining and Harnessing Adversarial Examples
• A Simple Framework for Contrastive Learning of Visual Representations
• Intriguing properties of neural networks
• Learning Transferable Visual Models From Natural Language Supervision
• Shortcut Learning in Deep Neural Networks
• Improved Baselines with Momentum Contrastive Learning
• Deep Learning in Spiking Neural Networks
• What Makes for Good Views for Contrastive Learning?
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• The Space of Transferable Adversarial Examples
• Deep Learning and its Application to LHC Physics
• Billion-scale semi-supervised learning for image classification
• Intriguing Properties of Vision Transformers
• Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
• Learning Robust Global Representations by Penalizing Local Predictive Power
• Learning with a Strong Adversary
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs
• Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks
• Image Synthesis with a Single (Robust) Classifier
• Interpreting Adversarially Trained Convolutional Neural Networks
• Adversarial Robustness as a Prior for Learned Representations
• Self-supervised learning through the eyes of a child
• Are Perceptually-Aligned Gradients a General Property of Robust Classifiers?
• Exposing Previously Undetectable Faults in Deep Neural Networks
• Are Convolutional Neural Networks or Transformers more like human vision?
• On the surprising similarities between supervised and self-supervised models
• The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
• Comparison-Based Framework for Psychophysics: Lab versus Crowdsourcing
• The shape and simplicity biases of adversarially robust ImageNet-trained CNNs
• Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)

Cited by in corpus (3)

• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
• DGMamba: Domain Generalization via Generalized State Space Model
• Trapped in texture bias? A large scale comparison of deep instance segmentation