Controversial stimuli: pitting neural networks against each other as models of human recognition

Abstract

Distinct scientific theories can make similar predictions. To adjudicate between theories, we must design experiments for which the theories make distinct predictions. Here we consider the problem of comparing deep neural networks as models of human visual recognition. To efficiently compare models' ability to predict human responses, we synthesize controversial stimuli: images for which different models produce distinct responses. We applied this approach to two visual recognition tasks, handwritten digits (MNIST) and objects in small natural images (CIFAR-10). For each task, we synthesized controversial stimuli to maximize the disagreement among models which employed different architectures and recognition algorithms. Human subjects viewed hundreds of these stimuli, as well as natural examples, and judged the probability of presence of each digit/object category in each image. We quantified how accurately each model predicted the human judgments. The best performing models were a generative Analysis-by-Synthesis model (based on variational autoencoders) for MNIST and a hybrid discriminative-generative Joint Energy Model for CIFAR-10. These DNNs, which model the distribution of images, performed better than purely discriminative DNNs, which learn only to map images to labels. None of the candidate models fully explained the human responses. Controversial stimuli generalize the concept of adversarial examples, obviating the need to assume a ground-truth model. Unlike natural images, controversial stimuli are not constrained to the stimulus distribution models are trained on, thus providing severe out-of-distribution tests that reveal the models' inductive biases. Controversial stimuli therefore provide powerful probes of discrepancies between models and human perception.

References in corpus (30)

• Scikit-learn: Machine Learning in Python
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• The NumPy array: a structure for efficient numerical computation
• Explaining and Harnessing Adversarial Examples
• Shortcut Learning in Deep Neural Networks
• On Calibration of Modern Neural Networks
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway
• Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition
• Delving into Transferable Adversarial Examples and Black-box Attacks
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Bayesian Active Learning for Classification and Preference Learning
• EMNIST: an extension of MNIST to handwritten letters
• Generalisation in humans and deep neural networks
• Illuminating search spaces by mapping elites
• Adversarial Examples Are Not Bugs, They Are Features
• Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Towards the first adversarially robust neural network model on MNIST
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Adversarial Examples that Fool both Computer Vision and Time-Limited Humans
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• Humans can decipher adversarial images
• Capturing human categorization of natural images at scale by combining deep networks and cognitive models
• Excessive Invariance Causes Adversarial Vulnerability
• Detecting and Diagnosing Adversarial Images with Class-Conditional Capsule Reconstructions
• Deep Predictive Coding Network for Object Recognition
• DARCCC: Detecting Adversaries by Reconstruction from Class Conditional Capsules
• Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness
• Understanding the Limitations of Conditional Generative Models

Cited by in corpus (17)

• Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future
• Going in circles is the way forward: the role of recurrence in visual inference
• Incorporating Dynamic Semantics into Pre-Trained Language Model for Aspect-based Sentiment Analysis
• A neural network walks into a lab: towards using deep nets as models for human behavior
• Distinguishing mirror from glass: A 'big data' approach to material perception
• Testing the limits of natural language models for predicting human language judgments
• Modeling biological face recognition with deep convolutional neural networks
• Guiding Visual Attention in Deep Convolutional Neural Networks Based on Human Eye Movements
• If deep learning is the answer, then what is the question?
• Transforming Neural Network Visual Representations to Predict Human Judgments of Similarity
• A 7T fMRI dataset of synthetic images for out-of-distribution modeling of vision
• How does the primate brain combine generative and discriminative computations in vision?
• Score-Based Generative Classifiers
• The Effects of Image Distribution and Task on Adversarial Robustness
• Can you hear me $\text{now}$? Sensitive comparisons of human and machine perception
• Exposing Semantic Segmentation Failures via Maximum Discrepancy Competition
• Evaluating Cross-Subject and Cross-Device Consistency in Visual Fixation Prediction