Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition

Abstract

The primate visual system achieves remarkable visual object recognition performance even in brief presentations and under changes to object exemplar, geometric transformations, and background variation (a.k.a. core visual object recognition). This remarkable performance is mediated by the representation formed in inferior temporal (IT) cortex. In parallel, recent advances in machine learning have led to ever higher performing models of object recognition using artificial deep neural networks (DNNs). It remains unclear, however, whether the representational performance of DNNs rivals that of the brain. To accurately produce such a comparison, a major difficulty has been a unifying metric that accounts for experimental limitations such as the amount of noise, the number of neural recording sites, and the number trials, and computational limitations such as the complexity of the decoding classifier and the number of classifier training examples. In this work we perform a direct comparison that corrects for these experimental limitations and computational considerations. As part of our methodology, we propose an extension of "kernel analysis" that measures the generalization accuracy as a function of representational complexity. Our evaluations show that, unlike previous bio-inspired models, the latest DNNs rival the representational performance of IT cortex on this visual object recognition task. Furthermore, we show that models that perform well on measures of representational performance also perform well on measures of representational similarity to IT and on measures of predicting individual IT multi-unit responses. Whether these DNNs rely on computational mechanisms similar to the primate visual system is yet to be determined, but, unlike all previous bio-inspired models, that possibility cannot be ruled out merely on representational performance grounds.

35 pages, 12 figures, extends and expands upon arXiv:1301.3530

References in corpus (2)

• Learning Feature Hierarchies with Centered Deep Boltzmann Machines
• The Neural Representation Benchmark and its Evaluation on Brain and Machine

Cited by in corpus (32)

• Neural network models and deep learning - a primer for biologists
• Artificial neural networks for neuroscientists: A primer
• Explanatory models in neuroscience: Part 1 -- taking mechanistic abstraction seriously
• Explanatory models in neuroscience: Part 2 -- constraint-based intelligibility
• Toward Goal-Driven Neural Network Models for the Rodent Whisker-Trigeminal System
• Modeling Object Recognition in Newborn Chicks using Deep Neural Networks
• A deep learning theory for neural networks grounded in physics
• Beyond linear regression: mapping models in cognitive neuroscience should align with research goals
• Revealing structure components of the retina by deep learning networks
• Deep driven fMRI decoding of visual categories
• Recurrent neural network models for working memory of continuous variables: activity manifolds, connectivity patterns, and dynamic codes
• Human perception in computer vision
• Measuring and Understanding Sensory Representations within Deep Networks Using a Numerical Optimization Framework
• Feynman Machine: The Universal Dynamical Systems Computer
• Thalamocortical contribution to solving credit assignment in neural systems
• Dissimilarity learning via Siamese network predicts brain imaging data
• Learning Deep Temporal Representations for Brain Decoding
• Hierarchical nucleation in deep neural networks
• What takes the brain so long: Object recognition at the level of minimal images develops for up to seconds of presentation time
• Examining Representational Similarity in ConvNets and the Primate Visual Cortex
• Introducing the structural bases of typicality effects in deep learning
• Statistical Mechanics of Neural Processing of Object Manifolds
• Representation of White- and Black-Box Adversarial Examples in Deep Neural Networks and Humans: A Functional Magnetic Resonance Imaging Study
• Comparison Against Task Driven Artificial Neural Networks Reveals Functional Organization of Mouse Visual Cortex
• On Simplicity and Complexity in the Brave New World of Large-Scale Neuroscience
• A Useful Motif for Flexible Task Learning in an Embodied Two-Dimensional Visual Environment
• An Analysis of Deep Neural Networks with Attention for Action Recognition from a Neurophysiological Perspective
• Non-uniqueness phenomenon of object representation in modelling IT cortex by deep convolutional neural network (DCNN)
• A Rich Source of Labels for Deep Network Models of the Primate Dorsal Visual Stream
• A Deeper Look at the Unsupervised Learning of Disentangled Representations in $\beta$-VAE from the Perspective of Core Object Recognition
• A brain basis of dynamical intelligence for AI and computational neuroscience
• Inferring brain-computational mechanisms with models of activity measurements