Deep Residual Networks Learn the Geodesic Curve in the Wasserstein Space

Abstract

Recent studies revealed the mathematical connection between deep neural networks (DNNs) and dynamic systems. However, the specific dynamics that DNNs, especially deep residual networks (ResNets), tend to learn during training remain insufficiently characterized. To this end, we model the forward propagation of deep residual networks using continuity equations, in which the measure is conserved and infinite curves in the measure space connect the input distribution to the output one of a ResNet. We find ResNets with $L_2$ regularization attempt to learn the geodesic curve in the Wasserstein space, induced by the optimal transport map. Compared with plain networks, ResNets can better approximate the geodesic curve, which explains why ResNets can be optimized and generalize better. Numerical experiments show that the data tracks of a ResNet tend to be line-shaped in terms of the line-shape score, and the map learned by a ResNet is closer to the optimal transport map in terms of the optimal transport score. In a word, we conclude that ResNets learn the geodesic curve in the Wasserstein space and discretely engineer the data transformation in high-dimensional spaces.

35 pages, 16 figures

References in corpus (13)

• Gaussian Approximation Potentials: the accuracy of quantum mechanics, without the electrons
• Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanics
• DGM: A deep learning algorithm for solving partial differential equations
• Solving high-dimensional partial differential equations using deep learning
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Understanding the Role of Individual Units in a Deep Neural Network
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Highway and Residual Networks learn Unrolled Iterative Estimation
• Revisiting the Importance of Individual Units in CNNs via Ablation
• Width Provably Matters in Optimization for Deep Linear Neural Networks
• Tessellated Wasserstein Auto-Encoders