Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex

Abstract

We discuss relations between Residual Networks (ResNet), Recurrent Neural Networks (RNNs) and the primate visual cortex. We begin with the observation that a special type of shallow RNN is exactly equivalent to a very deep ResNet with weight sharing among the layers. A direct implementation of such a RNN, although having orders of magnitude fewer parameters, leads to a performance similar to the corresponding ResNet. We propose 1) a generalization of both RNN and ResNet architectures and 2) the conjecture that a class of moderately deep RNNs is a biologically-plausible model of the ventral stream in visual cortex. We demonstrate the effectiveness of the architectures by testing them on the CIFAR-10 and ImageNet dataset.

arXiv commentThis version was written in Sept. 2016. For April 2016 version see v1 below

References in corpus (10)

• Striving for Simplicity: The All Convolutional Net
• On the Number of Linear Regions of Deep Neural Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Deep Networks with Stochastic Depth
• How to Construct Deep Recurrent Neural Networks
• Identity Mappings in Deep Residual Networks
• Learning Functions: When Is Deep Better Than Shallow
• Recurrent Convolutional Neural Networks for Scene Parsing
• How Important is Weight Symmetry in Backpropagation?
• How Deep is the Feature Analysis underlying Rapid Visual Categorization?

Cited by in corpus (61)

• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• Densely Connected Convolutional Networks
• Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
• Task-Driven Convolutional Recurrent Models of the Visual System
• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• Review: Deep Learning in Electron Microscopy
• Going in circles is the way forward: the role of recurrence in visual inference
• Learning Functions: When Is Deep Better Than Shallow
• Spiking Deep Residual Network
• Towards an integration of deep learning and neuroscience
• Residual Connections Encourage Iterative Inference
• Recurrent Batch Normalization
• Feedback Network for Image Super-Resolution
• A neural network walks into a lab: towards using deep nets as models for human behavior
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Monocular 3D Object Detection with Sequential Feature Association and Depth Hint Augmentation
• Deep Recurrent Architectures for Seismic Tomography
• Sharing Residual Units Through Collective Tensor Factorization in Deep Neural Networks
• Learning Implicitly Recurrent CNNs Through Parameter Sharing
• Spatially Adaptive Computation Time for Residual Networks
• Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
• Active Long Term Memory Networks
• Image Super-Resolution via Dual-State Recurrent Networks
• Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering
• Pre-training Graph Neural Networks with Kernels
• Can Active Memory Replace Attention?
• Hard Encoding of Physics for Learning Spatiotemporal Dynamics
• ThreshNet: An Efficient DenseNet Using Threshold Mechanism to Reduce Connections
• Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation
• Image Segmentation by Iterative Inference from Conditional Score Estimation
• Disentangling neural mechanisms for perceptual grouping
• A Way out of the Odyssey: Analyzing and Combining Recent Insights for LSTMs
• Convolutional Networks with Dense Connectivity
• Hierarchical Predictive Coding Models in a Deep-Learning Framework
• Transfer entropy-based feedback improves performance in artificial neural networks
• Stable and expressive recurrent vision models
• Selection dynamics for deep neural networks
• Robust neural circuit reconstruction from serial electron microscopy with convolutional recurrent networks
• Learning Normalized Inputs for Iterative Estimation in Medical Image Segmentation
• Gated Path Planning Networks
• Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermasks
• Deep learning for pedestrians: backpropagation in CNNs
• Compression of Deep Neural Networks for Image Instance Retrieval
• Learning Latent Causal Structures with a Redundant Input Neural Network
• Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks
• Learning compact generalizable neural representations supporting perceptual grouping
• A Dynamically Controlled Recurrent Neural Network for Modeling Dynamical Systems
• Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers
• One Size Fits Many: Column Bundle for Multi-X Learning
• Identity Connections in Residual Nets Improve Noise Stability
• Recurrent Connectivity Aids Recognition of Partly Occluded Objects
• Differential equations as models of deep neural networks
• Rethinking ResNets: Improved Stacking Strategies With High Order Schemes
• The Uncanny Similarity of Recurrence and Depth
• Towards WARSHIP: Combining Components of Brain-Inspired Computing of RSH for Image Super Resolution
• Iterative evaluation of LSTM cells
• Recurrent networks improve neural response prediction and provide insights into underlying cortical circuits
• Back-Projection Pipeline
• ItNet: iterative neural networks with small graphs for accurate, efficient and anytime semantic segmentation
• Adaptive and Iteratively Improving Recurrent Lateral Connections
• Review: Ordinary Differential Equations For Deep Learning