Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision

Abstract

Convolutional neural network (CNN) driven by image recognition has been shown to be able to explain cortical responses to static pictures at ventral-stream areas. Here, we further showed that such CNN could reliably predict and decode functional magnetic resonance imaging data from humans watching natural movies, despite its lack of any mechanism to account for temporal dynamics or feedback processing. Using separate data, encoding and decoding models were developed and evaluated for describing the bi-directional relationships be-tween the CNN and the brain. Through the encoding models, the CNN-predicted areas covered not only the ventral stream, but also the dorsal stream, albe-it to a lesser degree; single-voxel response was visualized as the specific pixel pattern that drove the response, revealing the distinct representation of individual cortical location; cortical activation was synthesized from natural images with high-throughput to map category representation, con-trast, and selectivity. Through the decoding models, fMRI signals were directly decoded to estimate the feature representations in both visual and semantic spaces, for direct visual reconstruction and seman-tic categorization, respectively. These results cor-roborate, generalize, and extend previous findings, and highlight the value of using deep learning, as an all-in-one model of the visual cortex, to understand and decode natural vision.

27 pages, 10 figures, 1 table

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Striving for Simplicity: The All Convolutional Net
• Unsupervised Learning of Video Representations using LSTMs
• Object Detectors Emerge in Deep Scene CNNs
• Deep Neural Networks predict Hierarchical Spatio-temporal Cortical Dynamics of Human Visual Object Recognition

Cited by in corpus (19)

• Neural network models and deep learning - a primer for biologists
• Controversial stimuli: pitting neural networks against each other as models of human recognition
• From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI
• Deep Predictive Coding Network for Object Recognition
• Deep Nets: What have they ever done for Vision?
• Accurate reconstruction of image stimuli from human fMRI based on the decoding model with capsule network architecture
• Using quantum states of light to probe the retinal network
• Revealing Fine Structures of the Retinal Receptive Field by Deep Learning Networks
• Discovering the Computational Relevance of Brain Network Organization
• A Neural Spiking Approach Compared to Deep Feedforward Networks on Stepwise Pixel Erasement
• Category decoding of visual stimuli from human brain activity using a bidirectional recurrent neural network to simulate bidirectional information flows in human visual cortices
• Towards the Next Generation of Retinal Neuroprosthesis: Visual Computation with Spikes
• Neural encoding with visual attention
• Predictive coding feedback results in perceived illusory contours in a recurrent neural network
• Deep learning approaches for neural decoding: from CNNs to LSTMs and spikes to fMRI
• Perceptual Deep Neural Networks: Adversarial Robustness through Input Recreation
• An encoding framework with brain inner state for natural image identification
• More Than Meets the Eye: Self-Supervised Depth Reconstruction From Brain Activity
• Adaptive Decoding via Hierarchical Neural Information Gradients in Mouse Visual Tasks