The Uncanny Similarity of Recurrence and Depth
arXiv:2102.11011
Abstract
It is widely believed that deep neural networks contain layer specialization, wherein neural networks extract hierarchical features representing edges and patterns in shallow layers and complete objects in deeper layers. Unlike common feed-forward models that have distinct filters at each layer, recurrent networks reuse the same parameters at various depths. In this work, we observe that recurrent models exhibit the same hierarchical behaviors and the same performance benefits with depth as feed-forward networks despite reusing the same filters at every recurrence. By training models of various feed-forward and recurrent architectures on several datasets for image classification as well as maze solving, we show that recurrent networks have the ability to closely emulate the behavior of non-recurrent deep models, often doing so with far fewer parameters.
References in corpus (15)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Understanding Neural Networks Through Deep Visualization
- AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
- EMNIST: an extension of MNIST to handwritten letters
- Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation
- High-Performance Large-Scale Image Recognition Without Normalization
- Deep Equilibrium Models
- Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
- Perceiver: General Perception with Iterative Attention
- DeepMath - Deep Sequence Models for Premise Selection
- Highway and Residual Networks learn Unrolled Iterative Estimation
- Trellis Networks for Sequence Modeling
- Can recursive neural tensor networks learn logical reasoning?
- Understanding Generalization through Visualizations
- Towards Proof Synthesis Guided by Neural Machine Translation for Intuitionistic Propositional Logic