Pixels to Voxels: Modeling Visual Representation in the Human Brain

Abstract

The human brain is adept at solving difficult high-level visual processing problems such as image interpretation and object recognition in natural scenes. Over the past few years neuroscientists have made remarkable progress in understanding how the human brain represents categories of objects and actions in natural scenes. However, all current models of high-level human vision operate on hand annotated images in which the objects and actions have been assigned semantic tags by a human operator. No current models can account for high-level visual function directly in terms of low-level visual input (i.e., pixels). To overcome this fundamental limitation we sought to develop a new class of models that can predict human brain activity directly from low-level visual input (i.e., pixels). We explored two classes of models drawn from computer vision and machine learning. The first class of models was based on Fisher Vectors (FV) and the second was based on Convolutional Neural Networks (ConvNets). We find that both classes of models accurately predict brain activity in high-level visual areas, directly from pixels and without the need for any semantic tags or hand annotation of images. This is the first time that such a mapping has been obtained. The fit models provide a new platform for exploring the functional principles of human vision, and they show that modern methods of computer vision and machine learning provide important tools for characterizing brain function.

Cited by in corpus (16)

• Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway
• Shallow and Deep Convolutional Networks for Saliency Prediction
• Deep Neural Networks predict Hierarchical Spatio-temporal Cortical Dynamics of Human Visual Object Recognition
• Adapting Deep Network Features to Capture Psychological Representations
• End-to-end Convolutional Network for Saliency Prediction
• Modeling Human Categorization of Natural Images Using Deep Feature Representations
• The Role of Typicality in Object Classification: Improving The Generalization Capacity of Convolutional Neural Networks
• Basic Level Categorization Facilitates Visual Object Recognition
• Deep driven fMRI decoding of visual categories
• Category decoding of visual stimuli from human brain activity using a bidirectional recurrent neural network to simulate bidirectional information flows in human visual cortices
• Learning Hierarchical Visual Representations in Deep Neural Networks Using Hierarchical Linguistic Labels
• Effective and efficient ROI-wise visual encoding using an end-to-end CNN regression model and selective optimization
• Do Deep Neural Networks Model Nonlinear Compositionality in the Neural Representation of Human-Object Interactions?
• Evaluating (and improving) the correspondence between deep neural networks and human representations
• Constraint-free Natural Image Reconstruction from fMRI Signals Based on Convolutional Neural Network
• Relating CNNs with brain: Challenges and findings