Implicit regularization of deep residual networks towards neural ODEs

Abstract

Residual neural networks are state-of-the-art deep learning models. Their continuous-depth analog, neural ordinary differential equations (ODEs), are also widely used. Despite their success, the link between the discrete and continuous models still lacks a solid mathematical foundation. In this article, we take a step in this direction by establishing an implicit regularization of deep residual networks towards neural ODEs, for nonlinear networks trained with gradient flow. We prove that if the network is initialized as a discretization of a neural ODE, then such a discretization holds throughout training. Our results are valid for a finite training time, and also as the training time tends to infinity provided that the network satisfies a Polyak-Lojasiewicz condition. Importantly, this condition holds for a family of residual networks where the residuals are two-layer perceptrons with an overparameterization in width that is only linear, and implies the convergence of gradient flow to a global minimum. Numerical experiments illustrate our results.

ICLR 2024 (spotlight). 40 pages, 3 figures

References in corpus (21)

• Gaussian Error Linear Units (GELUs)
• A Convergence Theory for Deep Learning via Over-Parameterization
• The Reversible Residual Network: Backpropagation Without Storing Activations
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• Dissecting Neural ODEs
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• DeepNet: Scaling Transformers to 1,000 Layers
• The Lipschitz Constant of Self-Attention
• Universal Approximation Property of Neural Ordinary Differential Equations
• Sinkformers: Transformers with Doubly Stochastic Attention
• Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning
• Generalization bounds for neural ordinary differential equations and deep residual networks
• The emergence of clusters in self-attention dynamics
• On Singular Value Inequalities for the Sum of Two Matrices
• Global Convergence of Gradient Descent for Deep Linear Residual Networks
• Do Residual Neural Networks discretize Neural Ordinary Differential Equations?
• On the infinite-depth limit of finite-width neural networks
• Global convergence of ResNets: From finite to infinite width using linear parameterization
• Scaling Properties of Deep Residual Networks
• On skip connections and normalisation layers in deep optimisation