Identity Connections in Residual Nets Improve Noise Stability

Abstract

Residual Neural Networks (ResNets) achieve state-of-the-art performance in many computer vision problems. Compared to plain networks without residual connections (PlnNets), ResNets train faster, generalize better, and suffer less from the so-called degradation problem. We introduce simplified (but still nonlinear) versions of ResNets and PlnNets for which these discrepancies still hold, although to a lesser degree. We establish a 1-1 mapping between simplified ResNets and simplified PlnNets, and show that they are exactly equivalent to each other in expressive power for the same computational complexity. We conjecture that ResNets generalize better because they have better noise stability, and empirically support it for both simplified and fully-fledged networks.

arXiv commentICML 2019 Workshop on Understanding and Improving Generalization in Deep Learning, additional analysis on a property called Dominant Gradient Flow of Residual Nets in Appendix D

References in corpus (7)

• Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
• Highway and Residual Networks learn Unrolled Iterative Estimation
• Skip Connections Eliminate Singularities
• Reversible Architectures for Arbitrarily Deep Residual Neural Networks
• Identity Matters in Deep Learning
• Multi-Residual Networks: Improving the Speed and Accuracy of Residual Networks
• The Loss Surface of Residual Networks: Ensembles and the Role of Batch Normalization