Task-Driven Convolutional Recurrent Models of the Visual System

Abstract

Feed-forward convolutional neural networks (CNNs) are currently state-of-the-art for object classification tasks such as ImageNet. Further, they are quantitatively accurate models of temporally-averaged responses of neurons in the primate brain's visual system. However, biological visual systems have two ubiquitous architectural features not shared with typical CNNs: local recurrence within cortical areas, and long-range feedback from downstream areas to upstream areas. Here we explored the role of recurrence in improving classification performance. We found that standard forms of recurrence (vanilla RNNs and LSTMs) do not perform well within deep CNNs on the ImageNet task. In contrast, novel cells that incorporated two structural features, bypassing and gating, were able to boost task accuracy substantially. We extended these design principles in an automated search over thousands of model architectures, which identified novel local recurrent cells and long-range feedback connections useful for object recognition. Moreover, these task-optimized ConvRNNs matched the dynamics of neural activity in the primate visual system better than feedforward networks, suggesting a role for the brain's recurrent connections in performing difficult visual behaviors.

NIPS 2018 Camera Ready Version, 16 pages including supplementary information, 6 figures

References in corpus (7)

• Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Systematic evaluation of CNN advances on the ImageNet
• Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
• Cortical microcircuits as gated-recurrent neural networks
• Feature-based Attention in Convolutional Neural Networks
• One-Shot Segmentation in Clutter

Cited by in corpus (19)

• Recurrence is required to capture the representational dynamics of the human visual system
• Artificial neural networks for neuroscientists: A primer
• Going in circles is the way forward: the role of recurrence in visual inference
• Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets
• Crowding Reveals Fundamental Differences in Local vs. Global Processing in Humans and Machines
• Neural Networks with Recurrent Generative Feedback
• TDAPNet: Prototype Network with Recurrent Top-Down Attention for Robust Object Classification under Partial Occlusion
• Unsupervised Learning from Video with Deep Neural Embeddings
• Disentangling neural mechanisms for perceptual grouping
• Stable and expressive recurrent vision models
• Identifying Learning Rules From Neural Network Observables
• Robust neural circuit reconstruction from serial electron microscopy with convolutional recurrent networks
• Convolutional Bipartite Attractor Networks
• Recurrent Feedback Improves Feedforward Representations in Deep Neural Networks
• A Neurally-Inspired Hierarchical Prediction Network for Spatiotemporal Sequence Learning and Prediction
• Epigenetic evolution of deep convolutional models
• Seeing in the dark with recurrent convolutional neural networks
• Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation
• Recurrent networks improve neural response prediction and provide insights into underlying cortical circuits