Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition

Abstract

The primate visual system achieves remarkable visual object recognition performance even in brief presentations and under changes to object exemplar, geometric transformations, and background variation (a.k.a. core visual object recognition). This remarkable performance is mediated by the representation formed in inferior temporal (IT) cortex. In parallel, recent advances in machine learning have led to ever higher performing models of object recognition using artificial deep neural networks (DNNs). It remains unclear, however, whether the representational performance of DNNs rivals that of the brain. To accurately produce such a comparison, a major difficulty has been a unifying metric that accounts for experimental limitations such as the amount of noise, the number of neural recording sites, and the number trials, and computational limitations such as the complexity of the decoding classifier and the number of classifier training examples. In this work we perform a direct comparison that corrects for these experimental limitations and computational considerations. As part of our methodology, we propose an extension of "kernel analysis" that measures the generalization accuracy as a function of representational complexity. Our evaluations show that, unlike previous bio-inspired models, the latest DNNs rival the representational performance of IT cortex on this visual object recognition task. Furthermore, we show that models that perform well on measures of representational performance also perform well on measures of representational similarity to IT and on measures of predicting individual IT multi-unit responses. Whether these DNNs rely on computational mechanisms similar to the primate visual system is yet to be determined, but, unlike all previous bio-inspired models, that possibility cannot be ruled out merely on representational performance grounds.

arXiv comment35 pages, 12 figures, extends and expands upon arXiv:1301.3530

References in corpus (2)

• Learning Feature Hierarchies with Centered Deep Boltzmann Machines
• The Neural Representation Benchmark and its Evaluation on Brain and Machine

Cited by in corpus (90)

• Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway
• STDP-based spiking deep convolutional neural networks for object recognition
• Neural network models and deep learning - a primer for biologists
• Generalisation in humans and deep neural networks
• Texture Synthesis Using Convolutional Neural Networks
• Artificial neural networks for neuroscientists: A primer
• Eigenvector Centrality Distribution for Characterization of Protein Allosteric Pathways
• Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner
• Comparing deep neural networks against humans: object recognition when the signal gets weaker
• Deep Networks Can Resemble Human Feed-forward Vision in Invariant Object Recognition
• Classification and Geometry of General Perceptual Manifolds
• The Origins and Prevalence of Texture Bias in Convolutional Neural Networks
• Controversial stimuli: pitting neural networks against each other as models of human recognition
• Going in circles is the way forward: the role of recurrence in visual inference
• Modeling the dynamics of human brain activity with recurrent neural networks
• Generic decoding of seen and imagined objects using hierarchical visual features
• Explanatory models in neuroscience: Part 1 -- taking mechanistic abstraction seriously
• Understanding trained CNNs by indexing neuron selectivity
• Deep Neural Networks predict Hierarchical Spatio-temporal Cortical Dynamics of Human Visual Object Recognition
• Linear Readout of Object Manifolds
• Parkinson's Disease Recognition Using SPECT Image and Interpretable AI: A Tutorial
• Unsupervised Learning on Neural Network Outputs: with Application in Zero-shot Learning
• Humans and deep networks largely agree on which kinds of variation make object recognition harder
• Emergence of functional and structural properties of the head direction system by optimization of recurrent neural networks
• Explanatory models in neuroscience: Part 2 -- constraint-based intelligibility
• CIFAR10 to Compare Visual Recognition Performance between Deep Neural Networks and Humans
• Photons guided by axons may enable backpropagation-based learning in the brain
• Modeling biological face recognition with deep convolutional neural networks
• Blindfold learning of an accurate neural metric
• Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces
• Deeper Interpretability of Deep Networks
• Toward Goal-Driven Neural Network Models for the Rodent Whisker-Trigeminal System
• The Topology and Geometry of Neural Representations
• Correspondence of Deep Neural Networks and the Brain for Visual Textures
• How can deep learning advance computational modeling of sensory information processing?
• Modeling Object Recognition in Newborn Chicks using Deep Neural Networks
• Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain
• Angular Visual Hardness
• Statistics of Visual Responses to Object Stimuli from Primate AIT Neurons to DNN Neurons
• The Role of Typicality in Object Classification: Improving The Generalization Capacity of Convolutional Neural Networks
• Guiding Visual Attention in Deep Convolutional Neural Networks Based on Human Eye Movements
• Basic Level Categorization Facilitates Visual Object Recognition
• A deep learning theory for neural networks grounded in physics
• Beyond linear regression: mapping models in cognitive neuroscience should align with research goals
• Revealing structure components of the retina by deep learning networks
• Deep driven fMRI decoding of visual categories
• On the relation between statistical learning and perceptual distances
• Recurrent neural network models for working memory of continuous variables: activity manifolds, connectivity patterns, and dynamic codes
• Human perception in computer vision
• Measuring and Understanding Sensory Representations within Deep Networks Using a Numerical Optimization Framework
• Finding Generalizable Evidence by Learning to Convince Q&A Models
• Thalamocortical contribution to solving credit assignment in neural systems
• Feynman Machine: The Universal Dynamical Systems Computer
• Dissimilarity learning via Siamese network predicts brain imaging data
• Teaching CNNs to mimic Human Visual Cognitive Process & regularise Texture-Shape bias
• Learning Deep Temporal Representations for Brain Decoding
• Compression of Deep Neural Networks on the Fly
• Statistical Mechanics of Neural Processing of Object Manifolds
• Introducing the structural bases of typicality effects in deep learning
• Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation
• What takes the brain so long: Object recognition at the level of minimal images develops for up to seconds of presentation time
• Hierarchical nucleation in deep neural networks
• Net2Brain: A Toolbox to compare artificial vision models with human brain responses
• Unsupervised learning on spontaneous retinal activity leads to efficient neural representation geometry
• Examining Representational Similarity in ConvNets and the Primate Visual Cortex
• Global Semantic Description of Objects based on Prototype Theory
• Diverse feature visualizations reveal invariances in early layers of deep neural networks
• Representation of White- and Black-Box Adversarial Examples in Deep Neural Networks and Humans: A Functional Magnetic Resonance Imaging Study
• ImageNet-trained deep neural network exhibits illusion-like response to the Scintillating Grid
• Comparison Against Task Driven Artificial Neural Networks Reveals Functional Organization of Mouse Visual Cortex
• Cognitive computational neuroscience
• Investigating Fine- and Coarse-grained Structural Correspondences Between Deep Neural Networks and Human Object Image Similarity Judgments Using Unsupervised Alignment
• Convolutional neural networks with extra-classical receptive fields
• Towards a Semantic Perceptual Image Metric
• Crowding in humans is unlike that in convolutional neural networks
• Deep Predictive Learning in Neocortex and Pulvinar
• On Simplicity and Complexity in the Brave New World of Large-Scale Neuroscience
• Model-Free Information Extraction in Enriched Nonlinear Phase-Space
• Supersaliency: A Novel Pipeline for Predicting Smooth Pursuit-Based Attention Improves Generalizability of Video Saliency
• Analogical and Relational Reasoning with Spiking Neural Networks
• Understanding Information Processing in Human Brain by Interpreting Machine Learning Models
• A brain basis of dynamical intelligence for AI and computational neuroscience
• A Deeper Look at the Unsupervised Learning of Disentangled Representations in $\beta$-VAE from the Perspective of Core Object Recognition
• Understanding learned CNN features through Filter Decoding with Substitution
• Non-uniqueness phenomenon of object representation in modelling IT cortex by deep convolutional neural network (DCNN)
• An Analysis of Deep Neural Networks with Attention for Action Recognition from a Neurophysiological Perspective
• The Computational Principles of Learning Ability
• A Rich Source of Labels for Deep Network Models of the Primate Dorsal Visual Stream
• A Useful Motif for Flexible Task Learning in an Embodied Two-Dimensional Visual Environment
• Inferring brain-computational mechanisms with models of activity measurements