Large-time asymptotics in deep learning

Abstract

We consider the neural ODE perspective of supervised learning and study the impact of the final time $T$ (which may indicate the depth of a corresponding ResNet) in training. For the classical $L^2$--regularized empirical risk minimization problem, whenever the neural ODE dynamics are homogeneous with respect to the parameters, we show that the training error is at most of the order $\mathcal{O}\left(\frac{1}{T}\right)$. Furthermore, if the loss inducing the empirical risk attains its minimum, the optimal parameters converge to minimal $L^2$--norm parameters which interpolate the dataset. By a natural scaling between $T$ and the regularization hyperparameter $\lambda$ we obtain the same results when $\lambda \searrow 0$ and $T$ is fixed. This allows us to stipulate generalization properties in the overparametrized regime, now seen from the large depth, neural ODE perspective. To enhance the polynomial decay, inspired by turnpike theory in optimal control, we propose a learning problem with an additional integral regularization term of the neural ODE trajectory over $[0,T]$. In the setting of ${\ell }^p$--distance losses, we prove that both the training error and the optimal parameters are at most of the order $\mathcal{O}\left(e^{-\mu t}\right)$ in any $t\in [0,T]$. The aforementioned stability estimates are also shown for continuous space-time neural networks, taking the form of nonlinear integro-differential equations. By using a time-dependent moving grid for discretizing the spatial variable, we demonstrate that these equations provide a framework for addressing ResNets with variable widths.

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Understanding Deep Convolutional Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• On Lazy Training in Differentiable Programming
• Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution
• Residual Flows for Invertible Generative Modeling
• Risk and parameter convergence of logistic regression
• Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity
• Turnpike Properties in Optimal Control: An Overview of Discrete-Time and Continuous-Time Results
• Stochastic Particle Gradient Descent for Infinite Ensembles
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory
• Universal Approximation Property of Neural Ordinary Differential Equations
• Network size and weights size for memorization with two-layers neural networks
• Momentum Residual Neural Networks
• A law of robustness for two-layers neural networks
• The turnpike property in semilinear control
• On the Turnpike to Design of Deep Neural Nets: Explicit Depth Bounds

Cited by in corpus (2)

• On the Turnpike to Design of Deep Neural Nets: Explicit Depth Bounds
• Quantitative estimates for parabolic optimal control problems under $L^{\infty }$ and $L^1$ constraints in the ball:Quantifying parabolic isoperimetric inequalities