Representing smooth functions as compositions of near-identity functions with implications for deep network optimization

Abstract

We show that any smooth bi-Lipschitz $h$ can be represented exactly as a composition $h_m\circ ...\circ h_1$ of functions $h_1,...,h_m$ that are close to the identity in the sense that each $\left(h_i-\mathrm{Id}\right)$ is Lipschitz, and the Lipschitz constant decreases inversely with the number $m$ of functions composed. This implies that $h$ can be represented to any accuracy by a deep residual network whose nonlinear layers compute functions with a small Lipschitz constant. Next, we consider nonlinear regression with a composition of near-identity nonlinear maps. We show that, regarding Fréchet derivatives with respect to the $h_1,...,h_m$, any critical point of a quadratic criterion in this near-identity region must be a global minimizer. In contrast, if we consider derivatives with respect to parameters of a fixed-size residual network with sigmoid activation functions, we show that there are near-identity critical points that are suboptimal, even in the realizable case. Informally, this means that functional gradient methods for residual networks cannot get stuck at suboptimal critical points corresponding to near-identity layers, whereas parametric gradient methods for sigmoidal residual networks suffer from suboptimal critical points in the near-identity region.

References in corpus (5)

• Gradient-based Hyperparameter Optimization through Reversible Learning
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Representation Benefits of Deep Feedforward Networks
• Learning Functions: When Is Deep Better Than Shallow
• Identity Matters in Deep Learning

Cited by in corpus (11)

• Data Driven Governing Equations Approximation Using Deep Neural Networks
• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
• GResNet: Graph Residual Network for Reviving Deep GNNs from Suspended Animation
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• A case for new neural network smoothness constraints
• Interpreting Deep Learning: The Machine Learning Rorschach Test?
• Are deep ResNets provably better than linear predictors?
• DeDUCE: Generating Counterfactual Explanations Efficiently
• Chaining Meets Chain Rule: Multilevel Entropic Regularization and Training of Neural Nets
• A class of robust numerical methods for solving dynamical systems with multiple time scales
• Step Size Matters in Deep Learning