Deep learning approaches for neural decoding: from CNNs to LSTMs and spikes to fMRI

Abstract

Decoding behavior, perception, or cognitive state directly from neural signals has applications in brain-computer interface research as well as implications for systems neuroscience. In the last decade, deep learning has become the state-of-the-art method in many machine learning tasks ranging from speech recognition to image segmentation. The success of deep networks in other domains has led to a new wave of applications in neuroscience. In this article, we review deep learning approaches to neural decoding. We describe the architectures used for extracting useful features from neural recording modalities ranging from spikes to EEG. Furthermore, we explore how deep learning has been leveraged to predict common outputs including movement, speech, and vision, with a focus on how pretrained deep networks can be incorporated as priors for complex decoding targets like acoustic speech or images. Deep learning has been shown to be a useful tool for improving the accuracy and flexibility of neural decoding across a wide range of tasks, and we point out areas for future scientific development.

arXiv comment22 pages, 3 figures

References in corpus (16)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Deep learning with convolutional neural networks for EEG decoding and visualization
• Axiomatic Attribution for Deep Networks
• SchNet - a deep learning architecture for molecules and materials
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• DeepSleepNet: a Model for Automatic Sleep Stage Scoring based on Raw Single-Channel EEG
• Deep Knowledge Tracing
• Sanity Checks for Saliency Maps
• Fast non-negative deconvolution for spike train inference from population calcium imaging
• CaloGAN: Simulating 3D High Energy Particle Showers in Multi-Layer Electromagnetic Calorimeters with Generative Adversarial Networks
• Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision
• Making brain-machine interfaces robust to future neural variability
• Deep learning as a tool for neural data analysis: speech classification and cross-frequency coupling in human sensorimotor cortex