Highway and Residual Networks learn Unrolled Iterative Estimation

Abstract

The past year saw the introduction of new architectures such as Highway networks and Residual networks which, for the first time, enabled the training of feedforward networks with dozens to hundreds of layers using simple gradient descent. While depth of representation has been posited as a primary reason for their success, there are indications that these architectures defy a popular view of deep learning as a hierarchical computation of increasingly abstract features at each layer. In this report, we argue that this view is incomplete and does not adequately explain several recent findings. We propose an alternative viewpoint based on unrolled iterative estimation -- a group of successive layers iteratively refine their estimates of the same features instead of computing an entirely new representation. We demonstrate that this viewpoint directly leads to the construction of Highway and Residual networks. Finally we provide preliminary experiments to discuss the similarities and differences between the two architectures.

arXiv comment10 + 4 pages, accepted for ICLR 2017

Cited by in corpus (58)

• Attention U-Net: Learning Where to Look for the Pancreas
• Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
• Tiny Machine Learning: Progress and Futures
• Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
• Review: Deep Learning in Electron Microscopy
• Going in circles is the way forward: the role of recurrence in visual inference
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Overcoming the vanishing gradient problem in plain recurrent networks
• Multi-level Residual Networks from Dynamical Systems View
• A Derivative-Free Method for Solving Elliptic Partial Differential Equations with Deep Neural Networks
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Rethinking Architecture Selection in Differentiable NAS
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning
• End-to-End Learning for Structured Prediction Energy Networks
• IamNN: Iterative and Adaptive Mobile Neural Network for Efficient Image Classification
• Learning Implicitly Recurrent CNNs Through Parameter Sharing
• Spatially Adaptive Computation Time for Residual Networks
• Greedy Layerwise Learning Can Scale to ImageNet
• SNODE: Spectral Discretization of Neural ODEs for System Identification
• Power-Enhanced Residual Network for Function Approximation and Physics-Informed Inverse Problems
• p-Meta: Towards On-device Deep Model Adaptation
• Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation
• ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation
• Cascaded Context Pyramid for Full-Resolution 3D Semantic Scene Completion
• Learning Deep Matrix Representations
• Predict and Constrain: Modeling Cardinality in Deep Structured Prediction
• Fast Neural Architecture Construction using EnvelopeNets
• SGAD: Soft-Guided Adaptively-Dropped Neural Network
• Graph Classification via Deep Learning with Virtual Nodes
• Generalization in Metric Learning: Should the Embedding Layer be the Embedding Layer?
• Differentiable Sparsification for Deep Neural Networks
• Sparsely Aggregated Convolutional Networks
• Comparing Fixed and Adaptive Computation Time for Recurrent Neural Networks
• FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Search
• Layer Folding: Neural Network Depth Reduction using Activation Linearization
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermasks
• DelugeNets: Deep Networks with Efficient and Flexible Cross-layer Information Inflows
• Shallow-to-Deep Training for Neural Machine Translation
• Why Layer-Wise Learning is Hard to Scale-up and a Possible Solution via Accelerated Downsampling
• Identity Connections in Residual Nets Improve Noise Stability
• Learning Light-Weight Translation Models from Deep Transformer
• On the Demystification of Knowledge Distillation: A Residual Network Perspective
• Exploring Weight Symmetry in Deep Neural Networks
• Relational dynamic memory networks
• Deep Learning: Our Miraculous Year 1990-1991
• The Uncanny Similarity of Recurrence and Depth
• Deep Residual Networks Learn the Geodesic Curve in the Wasserstein Space
• The Elastic Lottery Ticket Hypothesis
• Uncertainty Quantification in Deep Residual Neural Networks
• E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings
• Tandem Blocks in Deep Convolutional Neural Networks
• ItNet: iterative neural networks with small graphs for accurate, efficient and anytime semantic segmentation
• Residual networks classify inputs based on their neural transient dynamics
• OrthoReg: Robust Network Pruning Using Orthonormality Regularization
• Network Implosion: Effective Model Compression for ResNets via Static Layer Pruning and Retraining