Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex

Abstract

We discuss relations between Residual Networks (ResNet), Recurrent Neural Networks (RNNs) and the primate visual cortex. We begin with the observation that a special type of shallow RNN is exactly equivalent to a very deep ResNet with weight sharing among the layers. A direct implementation of such a RNN, although having orders of magnitude fewer parameters, leads to a performance similar to the corresponding ResNet. We propose 1) a generalization of both RNN and ResNet architectures and 2) the conjecture that a class of moderately deep RNNs is a biologically-plausible model of the ventral stream in visual cortex. We demonstrate the effectiveness of the architectures by testing them on the CIFAR-10 and ImageNet dataset.

This version was written in Sept. 2016. For April 2016 version see v1 below

References in corpus (3)

• Striving for Simplicity: The All Convolutional Net
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• How Deep is the Feature Analysis underlying Rapid Visual Categorization?

Cited by in corpus (59)

• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• Densely Connected Convolutional Networks
• Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
• Task-Driven Convolutional Recurrent Models of the Visual System
• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• Review: Deep Learning in Electron Microscopy
• Going in circles is the way forward: the role of recurrence in visual inference
• Spiking Deep Residual Network
• Towards an integration of deep learning and neuroscience
• Residual Connections Encourage Iterative Inference
• Feedback Network for Image Super-Resolution
• A neural network walks into a lab: towards using deep nets as models for human behavior
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Monocular 3D Object Detection with Sequential Feature Association and Depth Hint Augmentation
• Sharing Residual Units Through Collective Tensor Factorization in Deep Neural Networks
• Deep Recurrent Architectures for Seismic Tomography
• Learning Implicitly Recurrent CNNs Through Parameter Sharing
• Spatially Adaptive Computation Time for Residual Networks
• Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
• Active Long Term Memory Networks
• Image Super-Resolution via Dual-State Recurrent Networks
• Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering
• Pre-training Graph Neural Networks with Kernels
• Can Active Memory Replace Attention?
• Hard Encoding of Physics for Learning Spatiotemporal Dynamics
• ThreshNet: An Efficient DenseNet Using Threshold Mechanism to Reduce Connections
• Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation
• Image Segmentation by Iterative Inference from Conditional Score Estimation
• A Way out of the Odyssey: Analyzing and Combining Recent Insights for LSTMs
• Disentangling neural mechanisms for perceptual grouping
• Convolutional Networks with Dense Connectivity
• Hierarchical Predictive Coding Models in a Deep-Learning Framework
• Transfer entropy-based feedback improves performance in artificial neural networks
• Stable and expressive recurrent vision models
• Selection dynamics for deep neural networks
• Robust neural circuit reconstruction from serial electron microscopy with convolutional recurrent networks
• Gated Path Planning Networks
• Learning Normalized Inputs for Iterative Estimation in Medical Image Segmentation
• Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermasks
• Compression of Deep Neural Networks for Image Instance Retrieval
• Deep learning for pedestrians: backpropagation in CNNs
• Learning Latent Causal Structures with a Redundant Input Neural Network
• One Size Fits Many: Column Bundle for Multi-X Learning
• Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers
• Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks
• Learning compact generalizable neural representations supporting perceptual grouping
• A Dynamically Controlled Recurrent Neural Network for Modeling Dynamical Systems
• Rethinking ResNets: Improved Stacking Strategies With High Order Schemes
• Differential equations as models of deep neural networks
• Recurrent Connectivity Aids Recognition of Partly Occluded Objects
• Identity Connections in Residual Nets Improve Noise Stability
• Towards WARSHIP: Combining Components of Brain-Inspired Computing of RSH for Image Super Resolution
• The Uncanny Similarity of Recurrence and Depth
• Iterative evaluation of LSTM cells
• Review: Ordinary Differential Equations For Deep Learning
• Back-Projection Pipeline
• Adaptive and Iteratively Improving Recurrent Lateral Connections
• ItNet: iterative neural networks with small graphs for accurate, efficient and anytime semantic segmentation
• Recurrent networks improve neural response prediction and provide insights into underlying cortical circuits