Residual Networks Behave Like Ensembles of Relatively Shallow Networks

Abstract

In this work we propose a novel interpretation of residual networks showing that they can be seen as a collection of many paths of differing length. Moreover, residual networks seem to enable very deep networks by leveraging only the short paths during training. To support this observation, we rewrite residual networks as an explicit collection of paths. Unlike traditional models, paths through residual networks vary in length. Further, a lesion study reveals that these paths show ensemble-like behavior in the sense that they do not strongly depend on each other. Finally, and most surprising, most paths are shorter than one might expect, and only the short paths are needed during training, as longer paths do not contribute any gradient. For example, most of the gradient in a residual network with 110 layers comes from paths that are only 10-34 layers deep. Our results reveal one of the key characteristics that seem to enable the training of very deep networks: Residual networks avoid the vanishing gradient problem by introducing short paths which can carry gradient throughout the extent of very deep networks.

NIPS 2016

References in corpus (2)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improving neural networks by preventing co-adaptation of feature detectors

Cited by in corpus (90)

• Ensemble deep learning: A review
• Scaling Laws for Neural Language Models
• Deep Learning for Single Image Super-Resolution: A Brief Review
• AMP-Inspired Deep Networks for Sparse Linear Inverse Problems
• Machine Learning and Deep Learning -- A review for Ecologists
• Deep-Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks
• Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
• RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentation
• Simulator-free Solution of High-Dimensional Stochastic Elliptic Partial Differential Equations using Deep Neural Networks
• Path-Level Network Transformation for Efficient Architecture Search
• Cascaded Region-based Densely Connected Network for Event Detection: A Seismic Application
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• Approximating Continuous Functions by ReLU Nets of Minimal Width
• Review: Deep Learning in Electron Microscopy
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inference
• On the Origin of Deep Learning
• Automatic Liver Lesion Detection using Cascaded Deep Residual Networks
• ChemGrapher: Optical Graph Recognition of Chemical Compounds by Deep Learning
• Multi-level Residual Networks from Dynamical Systems View
• T-former: An Efficient Transformer for Image Inpainting
• Deep Hyperspherical Learning
• Gradient Boosting Neural Networks: GrowNet
• Real-time Semantic Image Segmentation via Spatial Sparsity
• Driver Behavior Recognition via Interwoven Deep Convolutional Neural Nets with Multi-stream Inputs
• Gait recognition via deep learning of the center-of-pressure trajectory
• Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey
• Deep Convolutional Neural Network Design Patterns
• Sparse Unsupervised Capsules Generalize Better
• Improving Multi-Scale Aggregation Using Feature Pyramid Module for Robust Speaker Verification of Variable-Duration Utterances
• SpotTune: Transfer Learning through Adaptive Fine-tuning
• BlockDrop: Dynamic Inference Paths in Residual Networks
• Multi-Residual Networks: Improving the Speed and Accuracy of Residual Networks
• PolyNet: A Pursuit of Structural Diversity in Very Deep Networks
• Pixel-wise Attentional Gating for Parsimonious Pixel Labeling
• All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation
• Analyzing the Real-World Applicability of DGA Classifiers
• Text to Image Synthesis Using Generative Adversarial Networks
• A Novel Weight-Shared Multi-Stage CNN for Scale Robustness
• Learning Deep ResNet Blocks Sequentially using Boosting Theory
• Sharing Residual Units Through Collective Tensor Factorization in Deep Neural Networks
• Data-Driven Sparse Structure Selection for Deep Neural Networks
• Deep Pyramidal Residual Networks
• IamNN: Iterative and Adaptive Mobile Neural Network for Efficient Image Classification
• DeepTracker: Visualizing the Training Process of Convolutional Neural Networks
• Model Slicing for Supporting Complex Analytics with Elastic Inference Cost and Resource Constraints
• Image Super-Resolution via Dual-State Recurrent Networks
• A Unified Deep Learning Framework for Short-Duration Speaker Verification in Adverse Environments
• Stochastic Training of Residual Networks: a Differential Equation Viewpoint
• Densely Connected High Order Residual Network for Single Frame Image Super Resolution
• Advancing System Performance with Redundancy: From Biological to Artificial Designs
• State Space Representations of Deep Neural Networks
• Residual Tensor Train: A Quantum-inspired Approach for Learning Multiple Multilinear Correlations
• Learning Transferable Adversarial Examples via Ghost Networks
• Single Image Super-resolution via a Lightweight Residual Convolutional Neural Network
• The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification
• NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding
• Improved Stereo Matching with Constant Highway Networks and Reflective Confidence Learning
• Recent Advances in the Applications of Convolutional Neural Networks to Medical Image Contour Detection
• Hard-Aware Deeply Cascaded Embedding
• Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex
• Long-Term Mobile Traffic Forecasting Using Deep Spatio-Temporal Neural Networks
• Learning spectro-temporal features with 3D CNNs for speech emotion recognition
• MDFN: Multi-Scale Deep Feature Learning Network for Object Detection
• Interpreting Deep Learning: The Machine Learning Rorschach Test?
• The Loss Surface of Residual Networks: Ensembles and the Role of Batch Normalization
• ZipNet-GAN: Inferring Fine-grained Mobile Traffic Patterns via a Generative Adversarial Neural Network
• Sparsely Aggregated Convolutional Networks
• White matter hyperintensity segmentation from T1 and FLAIR images using fully convolutional neural networks enhanced with residual connections
• FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Search
• Functional Gradient Boosting based on Residual Network Perception
• ResNetX: a more disordered and deeper network architecture
• On the Compressive Power of Deep Rectifier Networks for High Resolution Representation of Class Boundaries
• Two-Stream Binocular Network: Accurate Near Field Finger Detection Based On Binocular Images
• CrescendoNet: A Simple Deep Convolutional Neural Network with Ensemble Behavior
• Theoretical Analysis of the Advantage of Deepening Neural Networks
• On Residual Networks Learning a Perturbation from Identity
• Dynamical System Inspired Adaptive Time Stepping Controller for Residual Network Families
• Identification of images of COVID-19 from Chest Computed Tomography (CT) images using Deep learning: Comparing COGNEX VisionPro Deep Learning 1.0 Software with Open Source Convolutional Neural Networks
• ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
• Augmented Shortcuts for Vision Transformers
• ESFNet: Efficient Network for Building Extraction from High-Resolution Aerial Images
• An Exploration of Mimic Architectures for Residual Network Based Spectral Mapping
• cvpaper.challenge in 2016: Futuristic Computer Vision through 1,600 Papers Survey
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio
• Deep Competitive Pathway Networks
• Learning Robust and Adaptive Real-World Continuous Control Using Simulation and Transfer Learning
• SwGridNet: A Deep Convolutional Neural Network based on Grid Topology for Image Classification
• Is the Skip Connection Provable to Reform the Neural Network Loss Landscape?
• Bootstrapped CNNs for Building Segmentation on RGB-D Aerial Imagery