Identity Connections in Residual Nets Improve Noise Stability

Abstract

Residual Neural Networks (ResNets) achieve state-of-the-art performance in many computer vision problems. Compared to plain networks without residual connections (PlnNets), ResNets train faster, generalize better, and suffer less from the so-called degradation problem. We introduce simplified (but still nonlinear) versions of ResNets and PlnNets for which these discrepancies still hold, although to a lesser degree. We establish a 1-1 mapping between simplified ResNets and simplified PlnNets, and show that they are exactly equivalent to each other in expressive power for the same computational complexity. We conjecture that ResNets generalize better because they have better noise stability, and empirically support it for both simplified and fully-fledged networks.

ICML 2019 Workshop on Understanding and Improving Generalization in Deep Learning, additional analysis on a property called Dominant Gradient Flow of Residual Nets in Appendix D

References in corpus (3)

• Highway and Residual Networks learn Unrolled Iterative Estimation
• Reversible Architectures for Arbitrarily Deep Residual Neural Networks
• The Loss Surface of Residual Networks: Ensembles and the Role of Batch Normalization